Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Using system dynamics modelling to assess the economic efficiency of innovations in the public sector - a systematic review

  • Nidhee Jadeja ,

    Roles Conceptualization, Formal analysis, Methodology, Writing – original draft, Writing – review & editing

    n.jadeja18@imperial.ac.uk

    Affiliation NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance at Imperial College London, Imperial College London, London, United Kingdom

  • Nina J. Zhu,

    Roles Conceptualization, Formal analysis, Methodology, Writing – review & editing

    Affiliation NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance at Imperial College London, Imperial College London, London, United Kingdom

  • Reda M. Lebcir,

    Roles Conceptualization, Writing – review & editing

    Affiliation University of Hertfordshire Business School, Hatfield, United Kingdom

  • Franco Sassi,

    Roles Supervision, Writing – review & editing

    Affiliation Department of Economics & Public Policy, Centre for Health Economics & Policy Innovation, Imperial College Business School, South Kensington Campus, London, United Kingdom

  • Alison Holmes,

    Roles Supervision, Writing – review & editing

    Affiliation NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance at Imperial College London, Imperial College London, London, United Kingdom

  • Raheelah Ahmad

    Roles Conceptualization, Formal analysis, Methodology, Supervision, Writing – review & editing

    Affiliations NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance at Imperial College London, Imperial College London, London, United Kingdom, Division of Health Services Research and Management, School of Health Sciences, City, University of London, London, United Kingdom

Abstract

Background

Decision-makers for public policy are increasingly utilising systems approaches such as system dynamics (SD) modelling, which test alternative interventions or policies for their potential impact while accounting for complexity. These approaches, however, have not consistently included an economic efficiency analysis dimension. This systematic review aims to examine how, and in what ways, system dynamics modelling approaches incorporate economic efficiency analyses to inform decision-making on innovations (improvements in products, services, or processes) in the public sector, with a particular interest in health.

Methods and findings

Relevant studies (n = 29) were identified through a systematic search and screening of four electronic databases and backward citation search, and analysed for key characteristics and themes related to the analytical methods applied. Economic efficiency analysis approaches within SD broadly fell into two categories: as embedded sub-models or as cost calculations based on the outputs of the SD model. Embdedded sub-models within a dynamic SD framework can reveal a clear allocation of costs and benefits to periods of time, whereas cost calculations based on the SD model outputs can be useful for high-level resource allocation decisions.

Conclusions

This systematic review reveals that SD modelling is not currently used to its full potential to evaluate the technical or allocative efficiency of public sector innovations, particularly in health. The limited reporting on the experience or methodological challenges of applying allocated efficiency analyses with SD, particularly with dynamic embedded models, hampers common learning lessons to draw from and build on. Further application and comprehensive reporting of this approach would be welcome to develop the methodology further.

1. Introduction

Complex global problems such as climate change or antimicrobial resistance need innovation to shape impactful policies, systems, and services. As the current Covid-19 crisis reveals, the public sector plays a critical role in steering change to tackle the world’s most wicked problems; it is the only actor with the necessary legitimacy and resources to do so [1]. The world’s public sectors face acute fiscal and effectiveness pressures to tackle major challenges, it is therefore essential to ensure that policies represent good value for money. Innovation in the public sector refers to the implementation of a new or significantly changed product, which could be a good, service, or process, which can include production or delivery, organisation, and marketing processes [2]. Ex-ante simulation modelling of innovations in the health sector can help guide decision-makers, providing insight into how scenarios of different public sector innovations might play out in real-world settings. The notion of innovation has positive connotations attached to it, but a simulation model can reveal whether it creates any desired impacts or even possibly deleterious ones.

1.1. Using system dynamics to model complex public sector problems

Systems science approaches are increasingly being used to shape public sector innovations as they recognise [3, 4] the complexity of systems and mitigate the limitations of reductionist analytic modelling methods used to analyse these problems. Gault (2018) makes the case for a systems approach to analysing innovation in the public sector, recognising the potentially far-reaching impacts of actions beyond one specific sector and as a basis for developing more comprehensive policies [2]. In 2006 the United States Biomedical Advanced Research and Development Authority (BARDA) utilised a public health systems science approach to plan for pandemic influenza [5]. System Dynamics (SD) modelling is one such systems science approach that was originally developed in management science to represent and explain complex behaviours in a system such as patterns of non-linearity, externalities, and counterintuitive outcomes [6]. It uses computer simulation models to help address problems in complex systems and test alternative policies and scenarios in a systematic way [7]. SD tools such as causal loop diagrams (CLDs) and stock-flow diagrams (SFDs), are used to capture the non-linear mechanisms of a complex system [8]. These diagrammatic tools map the feedback structures and show how the system is dynamically influenced by the interactions of all variables [9].

Within the suite of systems methodologies, SD offers additional capabilities for informing intervention design and policy-making in comparison to soft systems methodologies by integrating qualitative and quantitative elements to represent soft behavioural variables, and engaging decision-makers in the process of testing policies or intervention strategies based on real-world circumstances [5, 10]. Qualitative approaches, including interviews and focus groups, can help elucidate key causal influences and factors in responding to a problem [11]. The participatory approach to model building in SD which engages stakeholders throughout the process ensures that real-world circumstances are taken into account. It enables organizational learning, aims to align stakeholder understanding of the underlying cause of and potential solutions to a problem, and facilitates consensus on the course of action and eventual policy adoption [12, 13]. SD has been gaining importance in informing health sector innovations as it can address common challenges in traditional approaches to policy-making, such as policy resistance, where actions triggered as a result of a policy undermine the policy or even exacerbate the original problem [1417]. Public sector resources however are finite and it is unclear how SD modelling has incorporated economic efficiency analyses, which provides crucial insight for policymakers in their decision-making.

1.2. The value of economic efficiency analyses for decision-making

Economists usually distinguish between two types of efficiency: technical and allocative efficiency. Technical efficiency refers to maximising activities or outcomes from a fixed set of resources, while allocative efficiency is concerned with directing resources to their most productive use to achieve the best overall benefits [18, 19]. Economic efficiency analyses, such as cost-effectiveness or cost-utility studies, compare options by their resource needs and subsequent benefits [20]. Economic efficiency analyses meaningfully contribute to health sector decisions by helping to set priorities and cost-effective plans, identifying the best ways of achieving strategic objectives, and providing insight on returns on investment [21]. Economic efficiency evaluations are an established practice in helping to inform public health sector decisions, however, on their own they typically represent a static snapshot of the situation rather than the shifting cost and benefit dynamics in the system over time [22, 23].

1.3. Aim and objectives

The aim of this systematic review was to examine and describe the range and nature of economic efficiency analyses in SD studies to understand how the shifting cost and benefit dynamics in the system have been evaluated for public sector innovations for complex problems. There was a particular interest for this review in the health sector, given the complex nature of health sector challenges and need for efficient use of resources. The specific objectives were to 1) Determine the policy target level (macro, meso, or micro level) for which the analysis has been conducted, 2) Compare approaches for how economic efficiency analyses have been incorporated with SD, and 3) Evaluate the quality and completeness of reporting of the economic efficiency analyses and SD modelling using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist and Rahmandad and Sterman’s guidelines for reporting for simulation-based research in social sciences [24, 25].

2. Method

A systematic search was conducted to identify articles published from 1st January 1999 to 1st June 2021 from the Scopus, Medline, EMBASE, Web of Science and Econlit databases using the search terms indicated in S1 Table. All the databases were last searched on the 10th June 2021. The review was limited to the past 22 years so as to reflect recent developments and current applications of SD. For both Medline and EMBASE, both keywords and MeSH terms were used. To further identify relevant articles, backward citation searches of two recent systematic reviews of SD modelling and health were conducted [17, 26, 27]. Only papers in English were included. Papers eligible for inclusion were those that described applications of an economic efficiency analysis in SD modelling to support a public sector decision-making process at any level of government and in any sector, to gain insights into the methodological approach itself. Studies excluded at both the title and abstract screening and full text screening were conference proceedings, those that did not use SD to assess the allocative efficiency between two or more policy options for a public sector, and those that were not available in the public domain. An example of a study that might appear to have met the inclusion criteria, but which was excluded is the study by Fontoura et al., which evaluated the impact of the existing Brazilian Urban Mobility Policy (BUMP) in the urban transport system, but did not involve an economic allocative efficiency analysis with SD to compare between two or more policy options [28]. Another example is the study by Lam and Mercure, which analyses which policy mixes are best for decarbonising passenger cars across five countries, assessing both the policies’ effectiveness in achieving emissions reductions and their cost-effectiveness in doing so [29]. This study was excluded as it did use System Dynamics modelling. Two reviewers (NJ and NJZ) independently screened all the study titles and abstracts and a third reviewer (CA) independently screened a randomised sample of 25% of the records using the software platform Covidence. Disparities were resolved through discussion to reach consensus. Following screening, two reviewers (NJ and NJZ) independently screened the full text of all manuscripts for inclusion into the review. The third reviewer (CA) independently screened a random sample of 25% of the studies. The detailed assessment of included studies was initially performed by one author (NJ) and reviewed by another author (RA). Once again, disparities were resolved through discussion and consensus. The reporting of this systematic review is in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (please see S2 Table for checklist) [30].

The quality and completeness of reporting of the economic efficiency analyses and SD modelling were examined in order to identify key themes related to the specific analysis methods applied, type of public sector innovation, and limitations associated with the approach. Although the specific economic efficiency analyses and overall model objectives of the studies vary significantly given the differing sectors, there are well-accepted guidelines for good modelling practices that the studies can be assessed against. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist, which provides guidance on good reporting practices in health economic evaluations, was adapted to assess the quality of reporting of the economic efficiency evaluations [24]. While the CHEERS checklist was developed for health economics evaluations specifically, it is largely applicable to economic evaluations more broadly, and thus was suitable as a benchmark/quality standard for which to assess the studies against. It includes analysis criteria such as target level of decision-making for policy, reporting of analytical methods, type of intervention or policy, limitations, and data sources. The quality and completeness of reporting of the SD model was assessed using Rahmandad and Sterman’s guidelines for reporting for simulation-based research in social sciences, which were developed to address the general lack of reporting guidelines in simulation-based research in the social sciences to facilitate the reproducibility of the simulation models [25]. The guidelines distinguish between factors essential for the reproduction of research and those that practice transparency, and only relevant guidelines were adapted and applied for this analysis and incorporated into the synthesis table referred to above. These include general visualisation guidelines, model reporting requirements, and simulation experiment reporting. Rahmandad and Sterman’s guidelines are more process-oriented and the CHEERS checklist is more focused on outcomes reporting; in this way they are a complementary set of criteria against which to evaluate the completeness of reporting of the studies.

The studies were also classified according to Windrum’s conceptualization of the six different types of public sector innovations, to understand where SD and economic evaluation has been applied [31]:

  1. Services innovation–described as “new or altered service features and design”
  2. Service delivery innovation–described as “new or altered ways of delivering services or interacting with citizens”
  3. Administrative or organisational innovation–described as “new or altered organizational methods in public sector practices, workplace organization or external relations, increasing public sector’s performance by reducing administrative/transaction costs, improve workplace satisfaction, etc.”
  4. Conceptual innovation–described as “the development of new world views that challenge assumptions that underpin existing service products, processes and organizational forms”
  5. Policy innovation–described as “to change the thought or behavioural intentions associated with a policy/new or altered missions, objectives, strategies and rationales”
  6. Systemic innovation–described as “new or improved ways of interacting with other organizations and knowledge bases”

The review protocol, template data collection forms, and data extracted are available upon request.

3. Results

3.1. Study selection

Fig 1 below summarises and describes the screening and checking process for final analysis and review. In total 6,608 records were identified through searching the databases. After removing duplicates and conducting backward citation searches, 4, 792 titles and abstracts were screened. From those screened, 101 full text articles were screened for eligibility and 29 studies were finally included for analysis.

3.2. General characteristics

Table 1 provides a summary of the general characteristics of the selected studies. Most of the studies (n = 19) were conducted in the latter half (2011–2021 of the review’s twenty-two year time period, which suggests an increasing need and recognition of SD to evaluate the cost aspects of public sector innovations. All the studies conducted a systematic analysis of the problem, and then used simulation to model to test the impacts of various innovation options.

3.3. Policy target level and geography

One of the aspects examined was the target level of the public sector innovation, which was classified as either macro (national), meso (regional, municipal), or micro (hospital, ICU unit) level (Fig 2). Seventy-six percent of the studies addressed innovations at the macro level, and four studies addressed innovations at the meso level. Ahmad et al. evaluated the policy of raising the legal smoking age at both the macro (i.e. in the USA) and meso (regional, i.e. in California) level through two separate published analyses, as tobacco control policies can be mandated at both the national and state levels in the USA [32, 33]. Three studies used economic efficiency analysis with SD to inform decision-making at the micro level. Mahmoudian-Dehkordi and Sadat (2017), for example, compared intensive care units (ICU) versus step-down or intermediate care unit (IMCU) capacity expansion in hospitals [34]. In terms of geography, twelve of the study settings were focused on the USA, with four studies focused on a low- and middle-income country setting.

3.4. Type of public sector and innovation

The studies used SD to inform decision-making across a number of different sectors, which included health (n = 20), transport (n = 3), climate (n = 1), water (n = 3), housing (n = 1), and energy (n = 1) sectors (Fig 3). Within health, the innovations being evaluated focused on both infectious and non-communicable diseases (NCDs), with Kivuti-Bitock’s evaluation of HPV vaccination and cervical cancer screening interventions in Kenya spanning both [35]. Countries around the world are facing increasing populations affected by ageing-associated diseases and conditions, and many of the studies in this review in the health sector examined the question decision-makers face regarding the balance between chronic disease prevention and management strategies. For example, Ansah et al. explored the health impact, costs, and cost savings of upstream and downstream interventions on the future number of chronic kidney disease and dialysis care patients in Singapore by 2040 [36]. Similarly, Sluijs et al. developed an SD model for policy-makers to understand and assess the impact and cost-effectiveness of lifestyle intervention programs on type 2 diabetes in the Netherlands [37]. A heterogenous mix of public sector innovation types were evaluated across the studies. According to Windrum’s typology, twenty studies evaluated policy innovations. These included a raising of the legal smoking age [32], reductions in CO2 emissions [38, 39], and regulations on groundwater aquifer use [40]. All of the studies evaluating services innovation (n = 2) or service delivery innovation (n = 11) or both (n = 1) were in the health sector. These included expanded provision for beds in intensive care units [34], increasing access to eye care services [41], screening and treatment services for breast cancer [42], and more intensive school-based anti-tobacco educational efforts [43].

Six of the studies sought to produce or use a generic or hypothetical simulation model in a specific public sector for future use by decision-makers. Alirezaei et al. for example, used SD to understand the complex interdependencies of the climate change-road safety-economy nexus itself, and develop a model platform that can be subsequently used by policymakers, rather than generate results of the model itself [38]. Duintjer Tebbens and Thompson model a hypothetical population in which two eradicable infectious diseases circulate, and evaluate different policy decision options on addressing them to show that cost-effectiveness decreases as the extent of priority-shifting increases [23]. Their study reveals how unintended consequences can arise from what might be considered intuitive decision rules in infectious disease control, and highlights the need to assess the costs and benefits of different policies when making decisions related to complex systems [ibid]. Three studies used a SD simulation model named The Prevention Impacts Simulation Model (PRISM) to explore the impacts of different interventions aimed at reducing cardiovascular disease. PRISM is a system dynamics model, originally developed in 2005 and funded by the US Centers for Disease Control and Prevention (CDC) and National Institutes of Health National Heart, Lung, and Blood Institute (NIH NHLBI), to simulate the health and cost outcomes for the entire US population from 1990 to 2040 and analyse the potential impacts of strategies to address cardiovascular disease risk factors [44]. The model reports summary measures of mortality and years of life lost and the medical and productivity costs of the chronic diseases and has been used by decision-makers at the local and federal level (ibid). Hirsch et al. used PRISM to explore the multiyear impacts of 22 different interventions aimed at reducing cardiovascular disease [45]. Yarnoff et al. used PRISM to simulate the potential 10-year and 25-year impact of clinical versus community interventions implemented by 32 communities in the United States, revealing the trade-offs decision-makers have to grapple with–clinical interventions had the potential to avert more premature deaths than community interventions, however, community interventions sustained over the long-term were more cost-effective [46]. Finally, Honeycutt et al. used PRISM to examine the potential cost-effectiveness of tobacco control changes implemented under a CDC-funded programme across 50 communities in the United States [47].

Other studies assessed specific innovations and compared the results to provide recommendations to decision-makers. Tejada et al., for example, compose discrete-event simulation (DES) and SD sub-models to evaluate the effectiveness of new or altered service delivery options for the screening and treatment of breast cancer in women 65+ in the USA [42]. Erten et al. compared targeted versus universal screening of colorectal cancers for Lynch Syndrome in terms of diagnostic accuracy and cost differences using real-world clinical data and not hypothetical assumptions [48].

3.5. Type of economic efficiency analysis

As shown in Table 1, the types of economic efficiency analyses conducted across the included studies were examined to better understand the capability of SD to incorporate different types. The most common type within the studies was cost-effectiveness analyses (CEA) of the innovations (n = 9), though the specific approaches to this varied. For example, Tejada et al. calculated the cost effectiveness ratio “average cost per quality adjusted life years (QALY) saved” in their breast cancer screening-and-treatment simulation for a ten year period using existing cost and QALY data from the US Department of Health and Human Services and academic literature [42]. Two studies conducted a cost-utility analysis (CUA). Kivuti-Bitock et al (2014) conducted a CUA, evaluating the effect of primary vaccination, secondary vaccination and screening campaigns for cervical cancer management in Kenya [35]. Evenden et al. developed a hybrid simulation model for dementia care services planning, showing that the currently available interventions of medication and a healthy lifestyle have only modest effects in terms of QALYs and reduced costs at the population level [49]. Disability Adjusted Life Years (DALYs), which represent the loss of the equivalent of one year of full health, and cost per averted DALY were used in the cost utility analysis [50]. DALYs consisted of Years of Life Lost (YLL) and Years of Life Lived with Disability (YLD), and the cost per averted DALY was based on a simplified calculation based on the total cost of intervention divided by the DALYs averted. Eight studies across the health, transport, water, housing, and energy sectors included cost-benefit analyses (CBA) [37, 5157]. MacMillan et al., for example, compared the effects of policy innovations to increase bicycle commuting in Auckland through a participatory SD approach. To ensure the policy relevance of their findings, they applied the New Zealand national transport agency’s methods to calculate indicative benefit-cost ratios for each policy scenario compared with baseline (summed net benefits divided by infrastructure costs) [51]. Evenden et al. developed a SD model for capturing Chlamydia infection dynamics within a population, and provide a cost-benefit study for required screening rates to manage infection prevalence [54].

Ten studies simply sought to assess the total cost impact of the innovations in question, though in these cases it was very important to understand how ‘cost’ was defined. Hirsch et al. for example, considered the effects of the interventions on deaths and downstream (or ‘consequence’) costs, with ‘costs’ referring to discounted (at 3% per year) direct medical costs for risk factor management and preventive care, acute care for CVD events and other risk factor-related hospitalizations, post-CVD long-term care, as well as productivity costs due to disability from CVD events and premature deaths from CVD events and other risk factor complications [45]. The costs did not include the administrative or non-medical implementation costs of interventions. In another study evaluating interventions for early childhood caries, Hirsch et al. report on both cumulative costs of restorative care and program costs as well as savings in restorative care compared to no intervention [58]. Data from dental offices and ambulatory or hospital sites was obtained to calculate the cost savings attributable to avoided restorative care from various interventions [ibid.]. Homer et al. avoided attempting to quantify intervention costs, which can be more difficult to estimate for broad classes of interventions through diverse strategies, and instead focused on consequence costs, arguing that this can still valuably help guide decision-makers by serving as a benchmark to justify the costs of interventions [59]. For example, if a given intervention results in a total consequence cost saving of $50 per capita, decision-makers can then justifiably spend up to $50 per capita for a given intervention and still maintain a net positive benefit [ibid.]. Consequence costs and savings were measured by medical and productivity (morbidity and mortality) costs using a human capital approach which estimates the market value of lost productivity at work and at home [ibid.].

CEA, which calculates a cost per unit of outcome for each intervention, and cost utility analysis (CUA), in which the incremental cost of an intervention or innovation is compared to the incremental benefit, are the most common forms of economic evaluation in health [60, 61]. Cost-effectiveness and cost-utility studies provide the investment case for choosing one innovation over another, but the focus on a single outcome can often limit its ability to capture the comprehensive range of costs and benefits [62]. CBA in contrast, synthesises and valuates all costs and benefits of an innovation in monetary units, and allows for a broader range of outcomes in monetary terms [61]. In theory this should allow for greater ease of comparison across innovations, but it is considered more vulnerable to bias in decision-making, as the included costs and benefits have to be measurable [60]. While none of the studies provided a rationale or justification for their selected approach, the different economic evaluation approaches conducted demonstrates the capability of SD to accommodate a wide range of types.

This review found that economic efficiency analysis approaches with SD within these studies broadly fell into two categories: as embedded sub-models or as cost calculations based on the outputs of the SD model. Smith and Ackere were motivated by the fact that decision-makers are often interested not just in the equilibrium predictions arising from an economic model, but also in the path taken by policy variables as they move towards equilibrium [63]. They demonstrate how it is possible to embed a simple static economic model within a dynamic SD framework, using the NHS as an example, to enhance the usefulness of the economic model [ibid]. Schade and Rothengatter take the rationale a step further, arguing that alternative approaches to traditional static models are needed for cost-benefit analyses [52]. They developed an SD platform that integrates a dynamic CBA of transport policies, revealing that the most favourable policy can change over time and depend on the time horizon defined for the analysis [ibid]. Their approach allows for a clear allocation of costs and benefits to periods of time, which may be particularly valuable for facilitating policy acceptance and implementation [ibid]. Milstein et al. also provide this type of dynamic temporal insight in their study of how the US system responds to large-scale interventions [64]. They demonstrate that while expanded health insurance coverage and better preventive and chronic care can save lives quickly, they tend to increase costs, and it is improved health behaviour and environmental conditions which are the critical ingredient over time for lowering both the number of deaths and reducing costs [ibid]. Ahmad on the other hand, used a SD simulation model to estimate smoking prevalence rates from policy changes to the legal smoking age, and then applied calculations of economic impacts to these outputs in terms of medical cost savings, cost of law enforcement, and cost of checking identification [32, 33].

3.6. Quality and completeness of reporting

Table 2 summarises the completeness of reporting of the economic efficiency analysis and SD modelling according to the CHEERS checklist and Rahmandad and Sterman’s guidelines in the selected studies, with a checkmark indicating where relevant information was provided. As can be seen, eighteen studies reported limitations and challenges however all of them related to the assumptions and estimations, rather than the technical aspects or application of economic efficiency analysis with SD or even to SD itself. The limited reporting on the experience or methodological challenges of applying cost analyses with SD, particularly with dynamic embedded models, hampers common learning lessons to draw from and build on. A Causal Loop Diagram (CLD), which is a visual representation of the dynamic relationships within a modelled system, is key to SD modelling, yet approximately a third of the studies (n = 6) did not include it in their publications. The CLDs are particularly informative in terms of understanding how economic aspects are positioned within dynamic relationships and how they influence them. Finally, a limited number of studies characterised the uncertainties of the economic analyses (n = 13) according to the CHEERS guidelines or reported on the statistical significance between policy scenarios in the overall SD models (n = 3) according to Rahmandad and Sterman’s guidelines. Characterising uncertainty enables decision-makers to better understand the information available, particularly in policy-making scenarios. While stakeholder input throughout the modelling process could arguably compensate for uncertainty, only five studies reported expert or stakeholder input and qualitative work as part of their SD approach. This is surprising, given expert or stakeholder engagement is a key feature of SD modelling and can help improve the predicting power of the model through assurance on whether the model is valid and representative of the real-world setting.

thumbnail
Table 2. Completeness of reporting of economic efficiency analysis and SD modelling in studies.

https://doi.org/10.1371/journal.pone.0263299.t002

4. Discussion

4.1 Key findings

While SD modelling is increasingly being used to examine complex public sector challenges, it is unclear how the methodology has incorporated economic efficiency analyses, which provides crucial insight for policymakers about trade-offs in resource-allocation in their decision-making. This systematic review of published studies therefore examined the range and nature of economic efficiency analyses in SD studies to understand how cost dynamics have been evaluated from a systems perspective for public sector problems. The first objective of this review was to determine the policy target level (macro, meso, or micro level) for which the analysis has been conducted. All of the studies were situated at one of the macro, meso, or micro but none explored interactions between different levels. As a review by Currie et al. noted, this represents a missed opportunity as most complex problems cross boundaries between micro, meso, and macro, and are rarely addressed at only one level [65].

The second objective of this review was to compare approaches for how economic efficiency analyses have been incorporated with SD. This review has found that the combined use of SD with economic efficiency analysis to evaluate public sector innovations has been increasing, including in the health sector. The majority of the studies were conducted within the last decade of the review period, and almost a quarter (n = 7) within the past eighteen months, indicating that the need for the combined use of SD with economic efficiency analysis has been increasing. Economic efficiency analysis within SD broadly fell into two categories: as embedded sub-models or as cost calculations based on the outputs of the SD model. The limitations described in the studies, primarily regarding the assumptions or estimations, are consistent with most types of simulation models and are not necessarily specific to the practice of combining economic efficiency analyses with SD.

The final objective of this review was to evaluate the quality and completeness of reporting of the economic efficiency analyses and SD modelling. The CHEERS checklist and Rahmandad and Sterman’s guidelines measure the quality and completeness of reporting rather than that of the underlying research, but these aspects are still very important, particularly for an emerging methodological approach, for reproducibility, and for influencing policy [24, 25]. A recent systematic review of SD applications in health and medicine more broadly, noted considerable shortcomings in model documentation, calibration, and validation in included publications, which is confirmed in this review [17].

SD is an iterative approach to policy analysis and design that recognises the complexity of problems, and is greatly strengthened when expert or stakeholder perspectives are included in the process. This has particularly valuable potential in the health sector where a diverse set of stakeholders can be involved. Most of the studies did not report on such consultations, and this represents a missed opportunity to strengthen the validity and credibility with decision-makers of the recommendations. There was little information presented in the publications on whether these studies influenced actual decisions or policies, or how effective they were which may not be possible given the time-lag of evidence to policy, but we recommend SD practitioners more explicitly report on whether policymakers were involved in the process.

4.2. Implications for policy-influence

The overall paucity of studies from multidisciplinary teams, however, suggests its full potential is not being met to support decision-making processes across a range of public sectors and geographies, particularly health policy. Most notably, no SD studies were found that examined the allocative efficiency of policy options in a Covid-19 context, despite the significant impact of the pandemic on human health and government budgets. Jay Wright Forrester, considered the founding father of SD, stated that “the failure of system dynamics to penetrate governments lies directly with the system dynamics profession and not with those in government” [65]. It is important for the SD community to facilitate making its methodology more mainstream and disseminating its contributions across disciplines, as results could help allow public sector decision-makers limited deploy limited resources better. In the process, its application prompts organizations to ask “what if?” questions, which can reveal the unforeseen implications of innovations. This is part of what is considered the purpose of the ‘anticipation’ dimension of Stilgoe et al.’s (2013) Framework for Responsible Innovation, which aims for more responsibility in the governance of emerging science and innovation through four integrated dimensions: anticipation, reflexivity, inclusion, and responsiveness [66]. Responsible innovation acknowledges that innovations can be unpredictable in terms of impacts, beneficial or otherwise and the use of SD to test alternative scenarios in a systematic way and assess their costs and benefits, can help ensure more responsible practice in the introduction of innovations in the public sector.

5. Limitations

This study relies on English-language studies available in the public domain and the use of SD terminology in the title or abstract, and it is therefore likely that some studies using SD may have been missed that are not published or used different terminology to describe their modelling approach. Furthermore, there may be debate over our selection framework, particularly the exclusion of applications of economic efficiency analyses with SD in the private sector.

6. Conclusion

The Covid-19 pandemic has highlighted the role and obligations the world’s public sectors face in tackling major challenges. Decision-makers have to grapple not only with ensuring the cost-effectiveness of policy measures during disruptions, but also ensure the protection of key sectors such as the health system or R&D ecosystem. Systems science modelling approaches combined with economic allocative efficiency analysis can play an important role in producing realistic evidence on policy options for decision-makers. There is a significant lack however in the scientific literature, assessing the economic allocative implications of policies in complex systems. This is the first systematic review examining the range and nature of economic efficiency analyses with SD methodologies for complex public policy problems. This review reveals that SD has a high applicability and demonstrated capability to evaluate the economic efficiencies of public sector innovations but is currently not utilized to its full potential to help decision-makers in developing effective actions. Future modelling studies should adhere more closely to good practice guidelines, in particular uncertainty and statistical significance analysis. Further application of this approach in the health sector would be welcome to develop the methodology further.

Supporting information

S1 Appendix. List of source articles in the systematic review.

https://doi.org/10.1371/journal.pone.0263299.s002

(DOCX)

Acknowledgments

The authors would like to thank Christine Agbenu for contributing as third reviewer to the study screening process.

References

  1. 1. Jordan A. and Huitema D., “Policy innovation in a changing climate: sources, patterns and effects.,” Global Environmental Change, vol. 19, pp. 387–394, 2014.
  2. 2. Gault F., “Defining and measuring innovation in all sectors of the economy. Research Policy.,” Research Policy, vol. 42, no. 47, pp. 617–622, 2018.
  3. 3. Greenhalgh T. and Papoutsi C., “Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Medicine.,” BMC Medicine, vol. 16, no. 1, p. 95, 2018. pmid:29921272
  4. 4. Rutter H., Savona N., Glonti K., Bibby J., Cummins S., Finegood D., et al. Thomas and M. White, “The need for a complex systems model of evidence for public health.,” Lancet, vol. 390, no. 10112, pp. 2602–2604, 2017. pmid:28622953
  5. 5. Atkinson J., Wells R., Page A., Dominello A., Haines M. and Wilson A., “Applications of system dynamics modelling to support health policy.,” Health Research Practice, vol. 25, no. 3, 2015. pmid:26243490
  6. 6. Society S. D., “What is SD? Introduction to System Dynamics.,” [Online]. Available: https://www.systemdynamics.org/what-is-sd. [Accessed 12 December 2019].
  7. 7. Forrester J., Industrial Dynamics, Cambridge: MIT Press, 1961.
  8. 8. Ahmad S., Tahar R., Muhammad-Sukki F., Munir A. and Rahim R., “Application of system dynamics approach in elecricity sector modelling: A review,” Renew. Sustain. Energy Rev., vol. 56, pp. 29–37, 2016.
  9. 9. Ding Z., Gong W., Li S. and Wu Z., “System dynamics versus agent-based modeling: a review of complexity simulation in construction waste management.,” Sustainability, vol. 10, no. 2484, 2018.
  10. 10. Maidstone R., “Discrete event simulation, system dynamics, and agent-based simulation: Discussion and comparison.,” 2012. [Online]. Available: https://pdfs.semanticscholar.org/75b3/096c6bd86eb1f946cacf8fde82fb37763c34.pdf?_ga=2.232021663.1870025104.1591881940-1394180601.1591881940. [Accessed 11 June 2020].
  11. 11. Sumari S., Ibrahim R. and Zakaria N., “Qualitative Analysis in System Dynamics for Health Care System.,” Journal of information systems research and innovation, vol. 6, 2014.
  12. 12. Homer J. and Hirsch G., “System dynamics modeling for public health: background and opportunities.,” Am J Public Health, vol. 96, no. 3, pp. 452–8, 2006. pmid:16449591
  13. 13. Rouwette E., Vennix J. and van Mullekom T., “Group model building effectiveness: a review of assessment studies.,” System Dynamics Review, vol. 18, no. 1, pp. 5–45, 2002.
  14. 14. Ghaffarzadegan N., Lyneis J. and Richardson G., “How small system dynamics models can help the public policy process.,” System Dynamics Review, vol. 27, no. 1, 2010.
  15. 15. Porter W., “The value of system dynamics modelling in policy analytics and planning.,” in Policy Analytics, Modelling, and Informatics, Springer, 2018.
  16. 16. Cavana R. and Clifford L., “Demonstrating the utility of system dynamics for public policy analysis in New Zealand: the case of excise tax policy on tobacco.,” System Dynamics Review, vol. 22, pp. 321–348, 2006.
  17. 17. Darabi N. and Hosseinichimeh N., “System dynamics modelling in health and medicine: a systematic literature review.,” System Dynamics Review, vol. 36, no. 1, 2020. pmid:32834468
  18. 18. Palmer S. and Torgerson D., “Economic notes: definitions of efficiency.,” BMJ, vol. 318, no. 7191, p. 1136, 1999. pmid:10213735
  19. 19. Shiell A., Donaldson C., Mitton C. and Currie G., “Health economic evaluation,” Journal of epidemiology and community health., vol. 56, pp. 85–88, 2002. pmid:11812804
  20. 20. National Institute for Health and Care Excellence, “Incorporating economic evaluation. Developing NICE guidelines: the manual.,” 31 October 2014. [Online]. Available: https://www.nice.org.uk/process/pmg20/chapter/incorporating-economic-evaluation. [Accessed 1 April 2020].
  21. 21. Manning M. and Johnson S., “Economic Analysis and Public Policy,” in Economic Analysis and Efficiency in Policing, Criminal Justice and Crime Reduction: What Works?, London, Palgrave Macmillan, 2016.
  22. 22. Nguyen T., Cook S. and Ireland V., “Application of system dynamics to evaluate the social and economic benefits of infrastructure projects.,” Systems, vol. 5, no. 29, 2017.
  23. 23. Duintjer Tebbens R. and Thompson K., “Priority shifting and the dynamics of managing eradicable infectious diseases.,” Management Science, vol. 55, no. 4, pp. 650–663, 2009.
  24. 24. Husereau D., Drummond M., Petrou S., Carswell C., Moher D., Greenberg D., et al. “Consolidated Health Economic Evaluation Reporting Standards (CHEERS) Statement.,” BMJ, vol. 346, no. f1049, 2013.
  25. 25. Rahmandad H. and Sterman J., “Reporting guidelines for simulation-based research in social sciences.,” System Dynamics Review, 2012.
  26. 26. Davahli M., Karwowski W. and Taiar R., “A system dynamics simulation applied to healthcare: a systematic review.,” International Journal of Environmental Research and Public Health, vol. 17, no. 5741, 2020. pmid:32784439
  27. 27. Higgins J., Lasserson T., Chandler J., Tovey D. and Churchill R., “Standards for the conduct of new Cochrane Intervention Reviews.,” in Methodological Expectations of Cochrane Intervention Reviews., London, Cochrane, 2016.
  28. 28. Fontoura W., Chaves G. and Ribeiro G., “The Brazilian urban mobility policy: The impact in Sao Paulo transport system using system dynamics,” Transport Policy, vol. 73, pp. 51–61, 2019.
  29. 29. Lam A. and Mercure J.-F., “Which policy mixes are best for decarbonising passenger cars? Simulating interactions among taxes, subsidies and regulations for the United Kingdom, the United States, Japan, China, and India,” Energy Research & Social Science, vol. 75, no. 101951, 2021.
  30. 30. Page M., McKenzie J., Bossuyt P., Boutron I., Hoffmann T., Mulrow C. and et al., “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.,” BMJ, vol. 372, no. n71, 2021.
  31. 31. Windrum P. and Koch P., Innovation and entrepreneurship in public services., Cheltenham: Edward Elgar, 2008. https://doi.org/10.1111/j.1365-2141.2008.07371.x pmid:18759759
  32. 32. Ahmad S., “The Cost-Effectiveness of Raising the Legal Smoking Age in California.,” Medical Decision Making, vol. 25, no. 3, pp. 330–340, 2005. pmid:15951460
  33. 33. Ahmad S., “Closing the youth access gap: The projected health benefits and cost savings of a national policy to raise the legal smoking age to 21 in the United States.,” Health Policy, vol. 75, no. 1, pp. 74–84, 2005. pmid:16298230
  34. 34. Mahmoudian-Dehkordi A. and Sadat S., “A Generic Simulation Model of the Relative Cost-Effectiveness of ICU Versus Step-down (IMCU) Expansion.,” Journal of Intensive Care Medicine, 2017. pmid:29088994
  35. 35. Kivuti-Block L., McDonnell G., Abdul R. and Pokhariyal G., “System Dynamics model of cervical cancer vaccination and screening interventions in Kenya.,” Cost effectiveness and resource allocation, vol. 12, no. 26, 2014.
  36. 36. Ansah J. P., Sheng Hng K. L., Ahmad S. and Goh C., “Evaluating the impact of upstream and downstream interventions on chronic kidney disease and dialysis care: a simulation analysis,” System Dynamics Review, vol. 37, no. 1, pp. 32–58, 2021.
  37. 37. Sluijs T., Lokkers L., Özsezen S., Veldhuis G. and Wortelboer H., “An Innovative Approach for Decision-Making on Designing Lifestyle Programs to Reduce Type 2 Diabetes on Dutch Population Level Using Dynamic Simulations.,” Front Public Health, vol. Apr, no. 29;9, p. 652694, 2021. pmid:33996729
  38. 38. Alirezaei M., Onat N., Tatari O. and Abdel-Aty M., “The climate change-road safety-economy nexus: A Systems Dynamics Approach to Understanding Complex Interdependencies.,” Systems, 2017.
  39. 39. Al-Foraih R., Sreekanth K., Alasseri R. and Al-Osaimi S., “A System Dynamic Energy Economic Assessment Model for Road Transportation,” International Energy Journal, vol. 20, no. Special Issue 3A, pp. 497–508, 2020.
  40. 40. Assaf H., “A hydro-economic model for managing groundwater resources in semi-arid regions.,” Water resources management, pp. 85–96, 2009.
  41. 41. Tuulonen A., Salminen H. and Linna M., “The need and total cost of Finnish eyecare services: a simulation model for 2005–2040.,” Acta Opthalmologica, vol. 87, no. 8, pp. 820–829, 2009. pmid:19740130
  42. 42. Tejada J., Ivy J., King R., Wilson J., Ballan M., Kay M., et al. “Combined DES/SD model of breast cancer screening for older women, II: screening-and-treatment simulation.,” IIE Transactions, vol. 46, pp. 707–727, 2014.
  43. 43. Tengs T., Osgood N. and Chen L., “The cost-effectiveness of intensive national school-based anti-tobacco education: results from the tobacco policy model.,” Prev Med, vol. 33, no. 6, pp. 558–570, 2001. pmid:11716651
  44. 44. Homer J., Wile K., Hirsch G., Trogdon J., Honeycutt A., Milstein B., et al. “Prevention Impacts Simulation Model (PRISM),” System Dynamics Society, 2021. [Online]. Available: https://systemdynamics.org/prism/. [Accessed 30th June 2021].
  45. 45. Hirsch G., Homer J., Trogdon J., Wile K. and Orenstein D., “Using Simulation to Compare 4 Categories of Interventions for Reducing Cardiovascular Disease Risks.,” American Journal of Public Health, vol. 104, no. 7, pp. 1187–1195, 2014. pmid:24832142
  46. 46. Yarnoff B., Bradley C., Honeycutt A., Soler R. and Orenstein D., “Estimating the Relative Impact of Clinical and Preventive Community-Based Interventions: An Example Based on the Community Transformation Grant Program.,” Prev Chronic Dis., vol. Jul 3, no. 16, p. E87, 2019. pmid:31274409
  47. 47. Honeycutt A., Bradley C., Khavjou O., Yarnoff B., Soler R. and Orenstein D., “Simulated impacts and potential cost effectiveness of Communities Putting Prevention to Work: Tobacco control interventions in 21 U.S. communities, 2010–2020,” Prev Med., vol. Mar, no. 120, pp. 100–106, 2019. pmid:30659909
  48. 48. Erten M., Fernandez L., Ng H., McKinnon W., Heald B., Koliba C., et al. “Universal Versus Targeted Screening for Lynch Syndrome: Comparing Ascertainment and Costs Based on Clinical Experience.,” Dig Dis Sci, vol. 61, no. 10, pp. 2887–2895, 2016. pmid:27384051
  49. 49. Evenden D., Brailsford S., Kipps C., Roderick P. and Walsh B., “Computer simulation of dementia care demand heterogeneity using hybrid simulation methods: improving population-level modelling with individual patient decline trajectories.,” Public Health., vol. Sep, no. 186, pp. 197–203, 2020. pmid:32861920
  50. 50. World Health Organization, “Disability Adjusted Life Years, Global Health Observatory Data.,” [Online]. Available: https://www.who.int/gho/mortality_burden_disease/daly_rates/text/en/. [Accessed 1 April 2020].
  51. 51. Macmillan A., Connor J., Witten K., Kearns R., Rees D. and Woodward A., “The Societal Costs and Benefits of Commuter Bicycling: Simulating the Effects of Specific Policies Using System Dynamics Modeling.,” Environmental Health Perspectives, vol. 122, no. 4, pp. 335–344, 2014. pmid:24496244
  52. 52. Schade W. and Rothengatter W., “Improving Assessment of Transport Policies by Dynamic Cost-Benefit Analysis.,” Transportation Research Record, pp. 107–114, 2015.
  53. 53. Shih Y. and Tseng C., “Cost-benefit analysis of sustainable energy development using life-cycle co-benefits assessment and the system dynamics approach.,” Applied Energy, vol. 119, pp. 57–66, 2014.
  54. 54. Evenden D., Harper PR, Brailsford S. and Harindra V., “System Dynamics modelling of Chlamydia infection for screening intervention planning and cost-benefit estimation.,” IMA Journal of Management Mathematics, vol. 16, pp. 265–279, 2005.
  55. 55. van Zyl A. and Jooste J. L., “Retaining and recycling water to address water scarcity in the City of Cape Town,” Development Southern Africa, no. Aug, 2020.
  56. 56. MacAskill S., Sahin O., Stewart R., Roca E. and Liu B., “Examining green affordable housing policy outcomes in Australia: A systems approach,” Journal of Cleaner Production, vol. 293, no. 126212, 2021.
  57. 57. Chen Y.-C., “System dynamics evaluation of household water use behavior and associated greenhouse gas emissions and environmental costs: A case study of Taipei city,” Journal of Water Process Engineering, vol. 37, no. 101409, 2020.
  58. 58. Hirsch G., Edelstein B., Frosh M. and Anselmo T., “A simulation model for designing effective interventions in early childhood caries.,” Prev. Chronic. Dis., vol. 9, no. 110219, 2012. pmid:22380939
  59. 59. Homer J., Milstein B., Wile K., Trogdon J., Huang P., Labarthe D., et al. “Simulating and evaluating local interventions to improve cardiovascular health.,” Chronic Dis, vol. 7, no. 1, p. A18, 2010. pmid:20040233
  60. 60. Perkins C., Steinbach R., Tompson L. and et al., “Chapter 4; cost-benefit analysis: methodological challenges of evaluating large-scale public health interventions and a worked example of the costs and benefits of part-night lighting.,” Sept 2015. [Online]. Available: https://www.ncbi.nlm.nih.gov/books/NBK316508/. [Accessed 1 April 2020].
  61. 61. Rai M. and Goyal R., “Pharmacoeconomics in Healthcare,” in Pharmaceutical medicine and translational clinical research, Academic Press, 2018, pp. 465–472.
  62. 62. Hoang V., Shanahan M., Shukla N., Perez P., Farrell M. and Ritter A., “A systematic review of modelling approaches in economic evaluations of health interventions for drug and alcohol problems.,” BMC Health Services Research, vol. 16, no. 127, 2016. pmid:27074871
  63. 63. Smith P. and Ackere A., “A note on the integration of system dynamics and economic models.,” Economic dynamics and control., vol. 26, pp. 1–10, 2002.
  64. 64. Milstein B., Homer J., Briss P., Burton D. and Pechacek T., “Why behavioral and environmental interventions are needed to improve health at lower cost.,” Health Aff, vol. 30, no. 5, pp. 823–832, 2011. pmid:21555468
  65. 65. Forrester J., “System dynamics—the next fifty years,” System Dynamics Review, vol. 23, no. 2–3, pp. 359–370, 2007.
  66. 66. Stilgoe J., Owen R. and Nacnaghten P., “Developing a framework for responsible innovation.,” Research Policy., vol. 42, pp. 1568–1580, 2013.