https://orcid.org/0000-0002-3872-3998
Figures
Abstract
Background
Whole disease models (WDM) are large-scale, system-level models which can evaluate multiple decision questions across an entire care pathway. Whilst this type of model can offer several advantages as a platform for undertaking economic analyses, the availability and quality of existing WDMs is unknown.
Objectives
This systematic review aimed to identify existing WDMs to explore which disease areas they cover, to critically assess the quality of these models and provide recommendations for future research.
Methods
An electronic search was performed on multiple databases (MEDLINE, EMBASE, the NHS Economic Evaluation Database and the Health Technology Assessment database) on 23rd July 2023. Two independent reviewers selected studies for inclusion. Study quality was assessed using the National Institute for Health and Care Excellence (NICE) appraisal checklist for economic evaluations. Model characteristics were descriptively summarised.
Results
Forty-four WDMs were identified, of which thirty-two were developed after 2010. The main disease areas covered by existing WDMs are heart disease, cancer, acquired immune deficiency syndrome and metabolic disease. The quality of included WDMs is generally low. Common limitations included failure to consider the harms and costs of adverse events (AEs) of interventions, lack of probabilistic sensitivity analysis (PSA) and poor reporting.
Conclusions
There has been an increase in the number of WDMs since 2010. However, their quality is generally low which means they may require significant modification before they could be re-used, such as modelling AEs of interventions and incorporation of PSA. Sufficient details of the WDMs need to be reported to allow future reuse/adaptation.
Citation: Jin H, Tappenden P, Ling X, Robinson S, Byford S (2023) A systematic review of whole disease models for informing healthcare resource allocation decisions. PLoS ONE 18(9): e0291366. https://doi.org/10.1371/journal.pone.0291366
Editor: Karina Cardoso Meira, Universidade Federal do Rio Grande do Norte, BRAZIL
Received: August 8, 2022; Accepted: August 28, 2023; Published: September 14, 2023
Copyright: © 2023 Jin et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The authors received no specific funding for this work. XL receives PhD scholarship (EP/R513143/1) from the Engineering and Physical Sciences Research Council (EPSRC), UK (https://www.ukri.org/councils/epsrc/). EPSRC had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The authors have no competing interests to declare that are relevant to the content of this article.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Health economic models are routinely used to inform healthcare resource allocation decisions in many countries across the world [1–5]. Models provide an explicit means of structuring a decision problem and synthesising all relevant evidence to estimate the expected costs and consequences of alternative health care interventions within a given health condition, usually over a lifetime horizon. Conventional health economic models are ‘piecewise’ in that they typically address a single decision problem at a specific decision point in a care pathway. ‘Piecewise’ models represent the standard analytic approach for informing decisions about the availability of health technologies by the National Institute for Health and Care Excellence (NICE) and similar agencies elsewhere [6–8], but they are subject to several limitations [9]. The first of these relates to the failure to capture system interdependencies between different interventions. The cost-effectiveness of any new intervention is dependent not only on the costs and effectiveness of the new intervention itself, but also on the configuration of the prevailing system, i.e. the availability, costs and effectiveness of existing interventions [9, 10]. For example, the cost-effectiveness of a new test for a given cancer type may be dependent on currently recommended treatment options for patients with diagnosed disease, as well as the availability of a screening programme for asymptomatic individuals. This type of system-level interdependency between interventions used for the same condition is seldom adequately captured by piecewise models due to their limited scope. Second, piecewise models often employ a simple piecewise cost per quality-adjusted life year (QALY) threshold rule which does not explicitly consider the budget constraint [11, 12]. However, it has been well documented that the repeated application of a threshold-based decision rule may lead to uncontrolled growth in health-care expenditure [13–18]. Third, most models are developed with the intention of informing a single decision problem within a broader pathway of care. This means that across a whole disease area, reimbursement and coverage decisions are based on a number of asynchronously developed discrete economic models which tend to apply different model structures, assumptions and evidence. This can lead to a situation whereby two models addressing the same decision question produce inconsistent conclusions, with potential to lead to sub-optimal adoption decisions [19–25].
System-level models, which include important events, health outcomes and costs across an entire disease area, represent a potential means of addressing the limitations of conventional piecewise models. Three well-known examples of system-level models include the US Archimedes diabetes model [26], the US Coronary Heart Disease (CHD) Policy Model [27], and the UK CHD model [28]. Although this type of modelling approach dates back to 1977 [29], it was not well-defined until 2012 when Tappenden et al. set out a methodological framework for whole disease models (WDM) [9]. In short, a WDM is a system-level generic disease model which allows for the consistent economic analysis of options across entire disease and treatment pathways, including prevention, detection, diagnosis and treatment [9]. Owing to the broader scope of these models, which focus on the whole disease and treatment pathway rather than individual decisions within that pathway, WDMs can provide a consistent conceptual and mathematical platform for the economic analysis of a large number of health care interventions based on a single model. In addition, WDMs can allow for the consideration of a variety of different economic decision rules which jointly deal with investment and disinvestment decisions through reference to a budget constraint, such as the disease-level constrained maximisation of health decision rule [30, 31]. Under this decision rule, different combinations of healthcare services (investment and disinvestment options) are tested [9]. The one that maximises health benefits while staying within the available budget is considered the most cost-effective choice.
Despite the potential benefits of the approach, the development of a WDM requires a significant initial investment of time and resources [9, 32] and presents additional challenges for model verification and validation. The initial investment in WDMs is therefore of greater value if they are re-used and adapted over time. However, re-use requires other modellers to be aware of existing WDMs and to determine their quality, as this will impact on whether the WDMs can be re-used. To our knowledge, no previous studies have systematically reviewed which WDMs exist, or the extent to which previously developed models could be re-used to address future decisions. To fill this gap, this systematic review aims to identify existing models meeting the criteria of a WDM for any disease (regardless of whether they were labelled as a WDM), to critically examine the quality of these models and to provide recommendations to improve the quality, reporting, and adaptability for future WDMs.
Methods
This systematic review was conducted according to the PRISMA recommendations for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions [33]. The protocol of this review was registered with PROSPERO (https://www.crd.york.ac.uk/prospero/) on 12th October 2020 (CRD42020199875).
Inclusion/exclusion criteria
Inclusion and exclusion criteria were defined a priori. It was hypothesised that models which strictly meet the criteria of a WDM defined by Tappenden et al [9] may be rare. Therefore, for the purpose of this systematic review, we decided to include studies which report models that broadly meet the criteria of being a WDM, i.e. a model which can evaluate multiple decision points (i.e. ≥3) for people with a given disease, and people at risk who may, or may not, go on to develop the health condition, and thus evaluates both the prevention and treatment of the condition simultaneously. Throughout this paper, these are referred to as ‘WDMs’, regardless of whether they were reported as a WDM in the original study. Those narrower models which evaluate multiple decision points (i.e., ≥3) only for people with a given disease (thus excluding people at risk of the disease), or only for people at risk of the disease (and thus excluding people with the given disease), are referred to as ‘pathways models’. These pathways models were excluded from the review as they do not meet our inclusion criteria of a WDM; however, a brief summary of identified pathways models is provided.
A model was assessed as meeting the criteria of a WDM either by demonstration (i.e., the authors used the model to address three or more decision points and reported the results in the paper) or based on authors’ reporting (i.e., the authors did not use the model to address evaluate options at multiple decision points in the paper, but they clearly reported that the model can be used in such a way). No limits were applied to the searches or the review inclusion criteria regarding specific diseases or conditions under consideration, types of economic evaluation, population, intervention or comparator, or outcome measures. Only published papers were included. Studies were excluded if they met any of the following criteria: (i) reviews, commentaries, letters, editorials, or abstracts; or (ii) not reported in English.
Whether a paper meets our inclusion criteria or not was determined based on the content of each individual paper, rather than the content of a series of related papers. Therefore, models which were used to address multiple decision points in a series of papers, but each paper only used that model to address one or two decision points were excluded.
Search strategy
Electronic biomedical databases searched included MEDLINE (including in-Process & other non-indexed) and EMBASE which were searched through the Ovid interface (https://ovidsp.ovid.com/). In addition, the NHS Economic Evaluation Database (NHS EED) and the Health Technology Assessment (HTA) Database were searched, accessed through the Cochrane library interface (http://onlinelibrary.wiley.com/cochranelibrary/search8). The search strategies included Medical Subject Heading (MeSH) terms and text words. Each follows a similar structure: economic evaluation-related terms AND WDM-related terms AND limitation terms about human studies and English language. The original searches and two update searches were conducted on 21 July 2020,18th July 2022, and 23rd July 2023 respectively. No restriction by publication year was applied. The full search strategy is reported in S1 Text. The reference lists of all identified WDMs were hand searched for further relevant studies which may have been missed by the electronic searches. Retrieved search results were downloaded into Endnote reference management software (Clarivate Analytics, version X9.3.3).
Assessment of abstracts and papers for inclusion
Screening of abstracts and papers against the inclusion criteria was undertaken independently by two reviewers (HJ and XL). Final inclusion of studies in the review was determined by agreement of both reviewers, with disagreements resolved by discussion.
Data extraction and analysis
Data were extracted by one reviewer (HJ) and checked by a second reviewer (XL), with disagreements resolved by discussion. The following information was extracted using Microsoft Excel (Microsoft Corporation, Microsoft 365 subscription) from all included studies: author; year; country; disease area; whether the model met the WDM or pathway model criteria by demonstration or authors’ reporting; number of decision points addressed; type of economic evaluation; main effectiveness outcome; modelling method; software; economic perspective; decision rule(s) used; affiliation of the corresponding author; and other information relevant for quality assessment. Study characteristics were summarised descriptively.
Quality assessment
Four commonly used checklists for economic evaluations were considered for the quality assessment of the current review, including the BMJ checklist (or Drummond checklist) [34], the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement [35], Philips’s checklist [36], and the NICE checklist [37]. Of these four checklists, the NICE checklist [37] was deemed to be most appropriate for the current review because (i) it focuses on the methodological quality of studies, as opposed to reporting quality (e.g. the CHEERS statement [35] focuses on the reporting quality rather than the methodological quality); (ii) it is appropriate for modelling studies, as opposed to trial-based economic evaluation (e.g. the BMJ checklist [34] is more appropriate for trial-based economic evaluation); and (iii) it allows the users to make an overall judgement regarding the methodological quality of the studies assessed. Therefore, it is easier to summarise and compare the methodological quality of a large number of included studies using the NICE checklist, compared to those checklists which do not provide an overall judgement regarding the methodological quality of the studies assessed (e.g. Philips’s checklist [36]).
The NICE checklist consists of two sections: Section 1 aims to assess the applicability of a study to the decision problems that need to be addressed, whilst Section 2 aims to assess the methodological quality of the study. Given that the aim of the review was to assess the methodological quality of the included study, rather than to assess the applicability of the model results to the UK setting, only Section 2 of the NICE checklist was used. Section 2 consists of twelve quality criteria and an overall assessment. Based on the number and importance of quality criteria that a study fails, an assessment regarding the overall methodological quality of the study can be classified into one of the following categories: (i) very serious limitations–the study fails to meet one or more quality criteria, and this is highly likely to change the conclusions about cost-effectiveness; (ii) potentially serious limitations–the study fails to meet one or more quality criteria, and this could change the conclusions about cost-effectiveness; and (iii) minor limitations–the study meets all quality criteria, or fails to meet one or more quality criteria but this is unlikely to change the conclusions about cost-effectiveness.
Two reviewers (HJ and XL) performed quality assessment for all included studies, with disagreements resolved by discussion.
Results
Study identification and selection
The detailed results of the literature search are reported in S1 Text. A modified preferred reporting items for systematic reviews and meta-analyses (PRISMA) diagram for the literature selection process is provided in Fig 1. A total of 7,090 citations were retrieved, with an additional 36 citations identified from checking the references of included studies. After removing duplicates, 4,803 citations remained. Of the 4,803 abstracts reviewed, 4,602 were excluded for clearly failing to meet at least one inclusion criterion or meeting at least one exclusion criterion, leaving 201 for full-text review. Of these, 41 were published only as abstracts and were excluded. Full texts of the remaining 160 citations were retrieved for detailed review. Of these, 40 papers reporting 44 WDMs satisfied the predefined inclusion criteria and were included in the review. The inter-reviewer agreement between HJ and XL, measured by Cohen’s kappa was 0.58, which indicates moderate agreement. A list of excluded studies with reasons are reported in S1 Table. A list of identified pathways models is reported in S2 Table.
Description of included WDMs
A brief summary of each included WDM is reported in Table 1. A summary of the key characteristics of the included WDMs are reported in Table 2. The most commonly modelled disease areas were heart disease (11/44, 25.0%), cancer (6/44, 13.6%), acquired immune deficiency syndrome (AIDS) (6/44, 13.6%) and metabolic disease (4/44, 9.1%). Of the 44 included WDMs, 33 (75.0%) met the criteria by demonstration and 11 (25.0%) met the criteria based on authors’ reporting. Eleven WDMs (11/44, 25.0%) were developed using PopMod and have very similar characteristics [38]. PopMod is a standard modelling tool developed by the WHO-CHOICE programme to facilitate disease modelling and cost-effectiveness analysis in diverse settings. The tool uses a multi-state dynamic life table method to simulate the evolution in time of an arbitrary population subject to births, deaths and two distinct disease conditions. Within PopMod, the default time horizon is lifetime and the default effectiveness outcome is disability-adjusted life years (DALYs). Of the eleven WDMs built using PopMod, eight addressed between 3 and 5 decision points (8/11, 72.7%) and three addressed between 6 and 10 decision points (3/11, 27.3%). Ten of the 11 PopMoD WDMs (90.9%) were published between 2010–2019. All eleven PopMoD WDMs were developed by non-commercial organisations. Five PopMoD WDMs (5/11, 45.5%) were developed for Sub-Saharan Africa and South East Asia [39–41], four (36.4%) were developed for Mexico [42] and two (18.2%) were developed for all 14 WHO regions [43, 44]. All eleven PopMoD WDMs used the piecewise cost per DALY threshold decision rule. None of the PopMod WDMs applied a disease-level constrained maximisation decision rule.
The characteristics of the remaining 33 WDMs varied greatly: for the sake of clarity, models which did not use the PopMod method are hereafter referred to as “other WDMs” throughout this paper. Eleven addressed less than 3 decision points but reported that they could address 3 or more (11/33, 33.3%), 19 addressed 3–5 decision points (57.6%) and 2 WDMs addressed over 10 decision points (6.1%) [30, 45]. These models were published between 1977 [29] to 2023 [46]. 66.7% of these WDMs (22/33) were published from 2010 onwards. The time horizon adopted ranged from 10-years to a lifetime. Twenty-one studies were cost-utility analyses (21/33, 63.6%), 11 studies were cost-effectiveness analyses (11/33, 33.3%) and one study was cost-benefit analysis (1/33, 3.0%) [47]. Sixteen studies used QALYs as the main effectiveness outcome (16/33, 48.5%), the rest used DALYs (6/33, 18.2%), disease incidence (5/33, 15.2%), life years (3/33, 9.1%), monetised health gains (1/33, 3.0%) [47], number of infection cases prevented (1/33, 3.0%) [46], and number of non-productive employees (1/33, 3.0%) [48]. The most commonly adopted modelling methods were cohort-level state transition model (10/33, 30.3%), individual-level state transition model (5/33, 15.2%), discrete event simulation (DES) (5/33, 15.2%), and dynamic compartmental model (5/33, 15.2%). The modelling software packages used were SIMUL8 (3/33, 9.1%), Excel (3/33, 9.1%), Python (3/33, 9.1%), TreeAge (2/33, 6.1%), R (2/33, 6.1%), Mathematica (2/33, 6.1%) and other (9/33, 27.3%). Eight WDMs did not report the modelling software used (8/33, 24.2%). Most of the WDMs were developed for high income countries, such as the USA (10/33, 30.3%), Netherlands (5/33, 15.2%), and the UK (4/33, 12.1%). All the WDMs compared competing options using a piecewise cost-effectiveness threshold-based decision rule; four of which (12.1%) also used disease-level constrained maximisation decision rule [31, 49–51]. Three of the WDMs were developed by commercial organisations [26, 45, 52] (3/33, 9.1%); the remainder were developed by non-commercial organisations (30/33, 90.9%).
Of all 44 identified WDMs, five reported that their models can be downloaded for free (11.4%) [47, 51, 53–55]. The other 39 WDMs (88.6%) did not report whether their model can be accessed by other researchers or not.
The complete evidence table for all included WDMs is reported in S3 Table.
Quality assessment
The results of the quality assessment for WDMs are summarised below; further detail is provided in the S4 Table. The reporting of most included WDMs was poor. Nineteen WDMs (19/44, 43.2%), including all eleven PopMod WDMs did not report details of their model structure or present their model structure diagrammatically. Fifteen WDMs (15/44, 34.1%) did not report the evidence sources used to inform resource use or unit cost inputs, and ten WDMs (10/44, 22.7%) did not report the evidence sources used to inform model parameters relating to clinical effectiveness of the technologies used within the treatment pathways.
According to the quality assessment results of the NICE checklist, of all 44 WDMs, 35 were deemed to have very serious limitations (79.5%), including all PopMoD models; five were deemed to have potentially serious limitations (11.4%) [47, 54, 55, 59, 71]; and four were deemed to have minor limitations (9.1%) [30, 31, 61, 65]. The performance of included PopMod WDMs (n = 11), other WDMs (n = 33), and all WDMs (n = 44) on all items of the NICE checklist is shown in Fig 2A–2C, respectively.
Of the PopMoD models, all of which were deemed to have very serious limitations, key problems identified included (Fig 2A):
- Inadequate model structure (11/11, 100%): PopMod is aimed at modelling initial disease treatment only and does not consider relapse or progress status; therefore, it is unlikely to capture the complexity of natural disease history or the entire care pathway.
- Failure to include all important and relevant outcomes and costs (11/11, 100%): the standard PopMod model only has four health states, including two groups with specific disease conditions, a group with the combined condition and a group with neither condition; as a result, PopMod has very limited capacity in modelling the health impacts of any adverse events (AEs) of interventions. None of the eleven PopMod WDMs in this review modelled any treatment-related AEs or justified their exclusion.
- Insufficient sensitivity analyses, for example, deterministic analysis for less than three parameters, or lack of probabilistic sensitivity analysis (PSA): PSA is useful for assessing the impact of joint uncertainty of multiple parameters simultaneously. However, the current version of PopMod does not support PSA and of the eleven PopMod WDMs included, only four of them conducted PSA [39, 41, 74, 75], and this required the use of an additional software package (MCLeague).
Of the 33 other WDMs, 24 were deemed to have very serious limitations (72.7%), five were deemed to have potentially serious limitations (15.2%), and four were deemed to have minor limitations (12.1%). Four key problems were identified (Fig 2B). First, twenty-nine WDMs (87.9%) failed to include all important and relevant outcomes and costs, for example, dis-utilities and costs resulting from AEs of interventions. Second, twenty-two WDMs (66.7%) failed to explore important uncertainties in their analysis using PSA or only conducted one-way sensitivity analyses for less than three parameters. Third, twenty WDMs (60.6%) did not report the source of resource use data or did not obtain resource use data from the best available source. For example, resource use data were estimated based on expert opinion or obtained from countries other than the one(s) of interest. Fourth, eighteen WDMs (54.5%) did not report their source of unit cost data or did not obtain unit cost data from the best available source. For example, unit cost data were estimated based on expert opinion or obtained from countries other than the one(s) of interest.
Discussion
Main findings and interpretation
Our review identified 44 WDMs. The first WDM was published as early as 1977 [29]. Over 70% of WDMs were published after 2010. The main disease areas covered by existing WDMs are heart disease, cancer, AIDS and metabolic disease, all of which are associated with significant disease burden. Only three WDMs (8%) were developed by commercial organisations [26, 45, 52]. This might be because most commercial companies are more likely to be motivated to develop piecewise models which evaluate a specific (type of) intervention, rather than developing WDMs which covers all interventions across the entire care pathway.
The majority of WDMs were of poor quality which means they may require significant modification before they can be re-used. These limitations included failure to consider the dis-utilities and cost caused by AEs of interventions, and lack of PSA. It was estimated that in OECD countries, 15% of total hospital activity and expenditure was a direct result of AEs [76]. Worldwide, the total disability-adjusted life years (DALYs) due to AEs of medical treatment was estimated to be 62.8 per 100,000 population in 2017 [77]. It has been reported that, for patients with schizophrenia [25], the cost-effectiveness of an intervention tends to be driven by its AE profile, rather than its clinical effectiveness. Given the potentially substantial impact of adverse events on cost-effectiveness results, many checklists and guidance for health economic analysis, such as the NICE checklist [37], the Philips’s checklist [36], Cooper’s hierarchy [78] and the ISPOR good practice guidance for budget impact analysis [79], all recommend that the dis-utilities and/or cost caused by AEs of interventions need to be considered in health economic analyses. PSA is mandated by many health technology assessment agencies, including the UK’s National Institute for Health and Care Excellence (NICE) [1] and the Canadian Agency for Drugs and Technologies in Health (CADTH) [80]. PSA is also recommended in the guidelines produced by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [81]. This is because for non-linear decision models (i.e. the relationship between the input data and the outcomes is not linear), PSA is required to provide a reliable estimate of mean costs and outcomes [1]. Given the complexity of entire disease pathways, WDMs are likely to be non-linear models. Therefore, being able to run PSA is an important function of a WDM.
Other limitations which may reduce the possibility or value of the WDMs for future use included (i) no reporting of whether their models can be accessed by other researchers or not; (ii) choice of modelling method and (iii) choice of modelling software. Of all WDMs included, only 23% used an individual-level modelling method. In order to evaluate multiple decision questions across the entire care pathway, a WDM often needs to incorporate a large number of events of interest and the relevant risk factors for all events of interest. If a cohort-level model was used to represent a large number of events and risk factors, thousands or even millions of health states will need to be built, which may place a huge burden on the model development and validation [82, 83]. Compared with cohort-level models, individual-level models might be more appropriate for developing a WDM, because they allow individual patient characteristics (e.g., risk factors for events of interest) and histories (e.g., events of interest) to be recorded [83–85]. In addition, individual-level models allow patients’ risk of events to change over time, depending on patient characteristics and previous events. Therefore, the methodological framework for developing WDMs [9] suggest that ‘given the level of depth and flexibility required to transfer the decision node across the model pathway, it is highly likely that individual-level simulation will be required.” One drawback of individual-level models is they require more computation time to run PSA compared with cohort-level models [86–88]. However, as computing power increases over time, the difference in computational burden between cohort-level and individual-level models reduces significantly [82, 83].
In terms of modelling software, our review found that a quarter of WDMs were developed using PopMoD. The main benefits of using PopMod are that such models are easy to build, free to use, allow modelling of one comorbidity, and support separate modelling of age and time [38]. In addition, the use of a standard modelling template facilitates comparison of results across different disease areas. However, PopMoD may not be the ideal tool in the context of building a WDM. Due to its simple structure with only four health states, PopMod does not allow modelling of disease relapse or progression or modelling of AEs, both of which are important considerations for a WDM which covers the entire natural disease history and a range of interventions. Other WDMs included in the review were developed using various modelling software, most of which are proprietary specialist software, such as Simul8, TreeAge and Matlab. Compared with open-source coding languages such as R and Python, specialist software might be easier to learn and use as they are equipped with pre-defined modules/functions. However, these software packages are often associated with a license fee.
Use of WDMs in clinical guidelines
By covering the entire care pathway and most key interventions within a single modelling framework, whole disease modelling can be used as an ideal foundation for economic evaluations in clinical guidelines. So far, whole disease modelling has been piloted on three published NICE clinical guidelines, covering three different disease areas: colorectal cancer [89], prostate cancer [90] and atrial fibrillation [91]. The reported advantages of using whole disease modelling for the selected NICE guidelines are: (a) it produces a considerably larger amount of economic information compared to traditional ‘piecewise’ models; (b) it improves consistency of cost-effectiveness results by using a single analytic framework and a common set of assumptions and data sources; (c) it can be used to compare the cost-effectiveness of interventions at different parts of the pathway (e.g. prevention and treatment), and explore system interdependencies between different interventions; and (d) once developed, WDMs can be reused to consider other related questions or to incorporate new evidence [30, 32].
The reported disadvantages of using whole disease modelling are mainly related to the technical and practical barriers. Large and complex models like WDMs are likely to be more prone to verification (programming) errors, more difficult to validate, and more difficult to explain to decision-makers than simple models. As a result, the development of each WDM has been estimated to take at least 12 months’ time of a full-time modeller. However, as Professor Alan Williams [92] pointed out, if creating such large-scale models is horrendously complicated, it is because reality is horrendously complicated; the more complex the reality is, the more dangerous it is to rely on intuitive short-cuts rather than careful analyses. Therefore, ‘more complex areas require models that respect complexity’ [84]. Given that WDMs, once developed, can be adapted and reused to provide ongoing support for decision-making across the entire care pathway, in the long-term, the benefits of using whole disease modelling in the context of clinical guidelines may outweigh the initial investment of time and money.
Recommendations for future research
Based on the findings of this review, there are a number of recommendations for future research:
- The reporting of future WDMs should follow the CHEERS statement [35]. In addition to the items listed in the CHEERS statement, we recommend that the following information should also be reported to help other researchers to decide whether the WDM can be reused/adapted to their settings: (1) software used to develop the model; and (2) a statement about access to the model, e.g. whether their model can be accessed by other researchers, and under what conditions. It is also recommended that a user manual is developed which includes a detailed description of each component of the model structure, all input data and instructions on how to use and/or adapt the model, as recommended by the STRESS guidelines for reporting models [93].
- It is recommended that for future WDMs, the appropriateness of alternative modelling methods should be assessed and the chosen method justified, this can be achieved by the application of model selection tools, such as the revised Brennan et al taxonomy [82].
- The health and cost impacts of important AEs of interventions should be considered for inclusion in the model and only excluded where these impacts are negligible and a clear justification is provided.
- Extensive sensitivity analyses need to be conducted to explore all key uncertainties in the model. As a minimum, one-way sensitivity analysis for all key parameters and PSA should be conducted.
In addition to the recommendation listed above, future research exploring methods for reducing the development time of WDMs is required to encourage greater use of the whole disease modelling approach. There are two solutions which could potentially help to reduce the development time of a WDM. The first solution is to adopt a team approach. The development of a WDM can be divided into several interrelated but different task modules, each of which requires a different set of skills (e.g., development of a conceptual model, identification and preparation of relevant input data, implementing the conceptual model within a computer software, and communicating the model results to stakeholders). This means the development work for a WDM can be potentially assigned to more than one researcher working in parallel, which would accelerate the development time of a WDM. However, the adoption of a team approach will lead to increased labour costs. The second solution is to develop templates for whole disease modelling. There are some similarities in terms of building health economic models for diseases of the same or similar types. For example, the modelling of cancer usually involves tumour progression from early stages to late stages, while the modelling of mental health problems usually involves repeated transitions between remission and relapse. Even for patients with different types of diseases, their QALYs accumulated at a specific stage of the care pathway usually depend on similar factors, such as: patient’s starting age, end age, current disease status, comorbidities and/or adverse events of interventions. The code for implementing these common functions can be made into a modelling template (or different templates for different types of disease) and published online. These templates may help to reduce the development time of future WDMs.
Strengths & limitations
Strengths.
To our knowledge, this is the first systematic review which outlines the availability and quality of WDMs for any disease areas. The information reported by this systematic review can be used to help researchers, commissioners or other stakeholders to rapidly locate relevant WDMs in the disease areas that they are interested in and critically appraise existing WDMs. Recommendations for future research can be used to fill evidence gaps and improve the quality and reporting of future WDMs.
Limitations.
This review is subject to at least five main limitations. Firstly, our search might not have identified all models which meet the criteria for a WDM. When designing the search strategy, we tried to include any search terms which might be relevant to a WDM, including terms such as “(full/comprehensive/entire/whole) adj3 (pathway*/system*/guideline*/disease*)”, “upstream adj2 downstream”, and “prevent* adj7 treat*”. However, unless we searched for all models for any diseases, there is no guarantee that all models meeting the criteria of a WDM have been identified. In addition, non-English literature and grey literature were not searched due to constraints in time, resources, and expertise within the reviewing team for non-English languages; and the inter-reviewer agreement is moderate (Cohen’s kappa = 0.58). The two reviewers (HJ and XL) tried to be more inclusive rather than exclusive when discussing any disagreed studies, however, there is still a possibility that some relevant WDMs have been missed by our review. Since the aim of this review was to provide an overview of existing WDMs, rather than to synthesise the results of the identified studies, we reckon the negative consequences of missing relevant studies are relatively small. Secondly, it should be noted that the definition of a WDM used for this review is more inclusive than the original definition provided in Tappenden et al. [9]. For example, Tappenden et al. defines a WDM as a model which includes the entire preclinical and post-diagnostic pathways for a given disease, allows the use of disease-level constrained optimisation, and allows decision node to be transferred across the modelled pathway. Since models meeting such criteria are rare, we decided to relax the definition of a WDM for this review to include models which can evaluate multiple decision points covering both the prevention and treatment of the disease simultaneously. This means not all WDMs included in this review cover the entire preclinical and post-diagnostic pathways for a given disease. As a result, not all included WDMs allow the use of disease-level constrained optimisation–of the 43 WDMs included, only four [31, 49–51] demonstrated that they allow the use of disease-level constrained optimisation. In addition, those included WDMs which were developed using a cohort-level modelling method are unlikely to allow a decision node to be transferred across the modelled pathway. Thirdly, whether a paper meets our inclusion criteria or not was determined based on the content of each individual paper, rather than the content of a series of related papers. Therefore, we might have missed those models which were used to address multiple decision points in a series of papers, but each paper only used that model to address one or two decision points. Fourthly, for those models which claimed they can be used to address three or more decision points but did not demonstrate this in the paper, whether they met the inclusion criteria of a WDM/pathway model or not was based on the authors’ reporting rather than our own assessment. Finally, we used the NICE checklist for economic evaluations [37] for assessing the quality of included WDMs. However, it should be noted that not all important aspects of quality assessment, such as model performance (e.g. comparing model results to real-world results) or model validation activities, were covered by the NICE checklist. In addition, the importance of quality criteria that a study fails (i.e. how likely this will change the conclusions about cost-effectiveness) was based on the reviewer’s judgement. Therefore, our results of quality assessment need to be interpreted with caution.
Conclusion
Despite their significant resource requirements associated with model development, there has been a significant increase in the number of WDMs since 2010. The main disease areas covered by existing WDMs are heart disease, cancer, metabolic disease and AIDS. A quarter of included WDMs were multi-state dynamic life table models developed using PopMoD; the remaining WDMs were developed using various modelling methods and software. The majority of WDMs were of poor quality which means they may require significant modification before they can be re-used, such as modelling AEs of interventions and incorporation of PSA. It is recommended that sufficient details of the WDMs need to be reported to allow future reuse/adaptation.
Supporting information
S1 Table. List of excluded studies with reasons.
https://doi.org/10.1371/journal.pone.0291366.s002
(XLSX)
S2 Table. Characteristics of included pathway models.
https://doi.org/10.1371/journal.pone.0291366.s003
(XLSX)
S3 Table. Characteristics of included whole disease models.
https://doi.org/10.1371/journal.pone.0291366.s004
(XLSX)
References
- 1.
National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. London, UK: National Institute for Health and Care Excellence; 2013.
- 2.
Scottish Medicines Consortium. Guidance to manufacturers for completion of new product assessment form (NPAF). Glasgow, UK: Scottish Medicines Consortium; 2007.
- 3.
Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (Version 4.5). Canberra, Australia: Pharmaceutical Benefits Advisory Committee; 2015.
- 4.
Canadian Agency for Drugs and Technologies in Health. Procedure and Submission Guidelines for the CADTH Common Drug Review. Ottawa, Canada: Canadian Agency for Drugs and Technologies in Health; 2019.
- 5. Luyten J, Naci H, Knapp M. Economic evaluation of mental health interventions: an introduction to cost-utility analysis. Evidence-based mental health. 2016;19(2):49–53. Epub 2016/04/15. pmid:27075444.
- 6. Raftery J. Review of NICE’s recommendations, 1999–2005. British Medical Journal. 2006;332(7552):1266–8. pmid:16735341
- 7. George B, Harris A, Mitchell A. Cost-effectiveness analysis and the consistency of decision making: evidence from pharmaceutical reimbursement in australia (1991 to 1996). Pharmacoeconomics. 2001;19(11):1103–9. Epub 2001/12/12. pmid:11735677.
- 8. Clement FM, Harris A, Li JJ, Yong K, Lee KM, Manns BJ. Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada. The Journal of the American Medical Association. 2009;302(13):1437–43. Epub 2009/10/08. pmid:19809025.
- 9. Tappenden P, Chilcott J, Brennan A, Squires H, Stevenson M. Whole Disease Modeling to Inform Resource Allocation Decisions in Cancer: A Methodological Framework. Value in Health. 2012;15(8):1127–36. pmid:23244816
- 10. Hauck K, Thomas R, Smith PC. Departures from Cost-Effectiveness Recommendations: The Impact of Health System Constraints on Priority Setting. Health Systems & Reform. 2016;2(1):61–70. pmid:31514655
- 11.
Drummond M, Sculpher MJ, Torrance GW, O’Brien BJ, G. S. Methods for the economic evaluation of health care programmes. 3rd ed. Oxford UK: Oxford University Press; 2005.
- 12. Culyer AJ. Cost-effectiveness thresholds in health care: a bookshelf guide to their meaning and use. Health Economics, Policy and Law. 2016;11(4):415–32. Epub 2016/02/24. pmid:26906561
- 13. Gafni A, Birch S. Guidelines for the adoption of new technologies: a prescription for uncontrolled growth in expenditures and how to avoid the problem. Canadian Medical Association Journal. 1993;148(6):913–7. Epub 1993/03/15. pmid:8448705; PubMed Central PMCID: PMC1490730.
- 14. Gafni A, Birch S. Inclusion of drugs in provincial drug benefit programs: Should "reasonable decisions" lead to uncontrolled growth in expenditures? Canadian Medical Association Journal. 2003;168(7):849–51. Epub 2003/04/02. pmid:12668543; PubMed Central PMCID: PMC151991.
- 15. Lord J, Laking G, Fischer A. Health care resource allocation: is the threshold rule good enough? Journal of health services research & policy. 2004;9(4):237–45. Epub 2004/10/29. pmid:15509410.
- 16. Birch S, Gafni A. The biggest bang for the buck or bigger bucks for the bang: the fallacy of the cost-effectiveness threshold. Journal of health services research & policy. 2006;11(1):46–51. Epub 2005/12/28. pmid:16378532.
- 17. Donaldson C, Currie G, Mitton C. Cost effectiveness analysis in health care: contraindications. British Medical Journal. 2002;325(7369):891–4. Epub 2002/10/19. pmid:12386045; PubMed Central PMCID: PMC1124387.
- 18.
Mitton C, Donaldson C. Priority-Setting Toolkit: A Guide to the Use of Economics in Healthcare Decision-Making. 1st ed. London, UK: BMJ Publishing Group; 2009.
- 19.
National Collaborating Centre for Mental Health. Psychosis and schizophrenia in adults: prevention and management. NICE guideline (CG178). London, UK: The British Psychological Society and The Royal College of Psychiatrists; 2014.
- 20. Achilla E, McCrone P. The cost effectiveness of long-acting/extended-release antipsychotics for the treatment of schizophrenia: A systematic review of economic evaluations. Applied health economics and health policy. 2013:95–106. Epub 2. doi: https://doi.org/https://doi.org/10.1007/s40258-013-0016-2. PubMed PMID: Peer Reviewed Journal: 2014-36824-002.
- 21. von Scheele B, Mauskopf J, Brodtkorb TH, Ainsworth C, Berardo CG, Patel A. Relationship between modeling technique and reported outcomes: case studies in models for the treatment of schizophrenia. Expert Review of Pharmacoeconomics & Outcomes Research. 2014;14(2):235–57. Epub 2014/02/26. pmid:24564639.
- 22. Zhou J, Millier A, Toumi M. Systematic review of pharmacoeconomic models for schizophrenia. J Mark Access Health Policy. 2018;6(1):1508272-. pmid:30128087.
- 23. Nemeth B, Fasseeh A, Molnar A, Bitter I, Horvath M, Koczian K, et al. A systematic review of health economic models and utility estimation methods in schizophrenia. Expert Rev Pharmacoecon Outcomes Res. 2018;18(3):267–75. Epub 2018/01/20. pmid:29347854.
- 24. Jin H, Tappenden P, Robinson S, Achilla E, MacCabe JH, Aceituno D, et al. A systematic review of economic models across the entire schizophrenia pathway. PharmacoEconomics. 2020;38(6):537–55. pmid:32144726
- 25. Jin H, Tappenden P, Robinson S, Achilla E, Aceituno D, Byford S. Systematic review of the methods of health economic models assessing antipsychotic medication for schizophrenia. PLOS ONE. 2020;15(7):e0234996. pmid:32649663
- 26. Schlessinger L, Eddy DM. Archimedes: a new model for simulating health care systems—the mathematical formulation. Journal of Biomedical Informatics. 2002;35(1):37–50. pmid:12415725
- 27. Weinstein MC, Coxson PG, Williams LW, Pass TM, Stason WB, Goldman L. Forecasting coronary heart disease incidence, mortality, and cost: the Coronary Heart Disease Policy Model. American journal of public health. 1987;77(11):1417–26. Epub 1987/11/01. pmid:3661794; PubMed Central PMCID: PMC1647098.
- 28. Davies R, Normand C, Raftery J, Roderick P, Sanderson C. Policy Analysis for Coronary Heart Disease: a Simulation Model of Interventions, Costs and Outcomes. 2003.
- 29. Cretin S. Cost/benefit analysis of treatment and prevention of myocardial infarction. Health services research. 1977;12(2):174–89. pmid:407178
- 30. Tappenden P, Chilcott J, Brennan A, Squires H, Glynne-Jones R, Tappenden J. Using whole disease modeling to inform resource allocation decisions: economic evaluation of a clinical guideline for colorectal cancer using a single model. Value in Health. 2013;16(4):542–53. Epub 2013/06/26. pmid:23796288.
- 31. Jin H, Tappenden P, MacCabe JH, Robinson S, Byford S. Evaluation of the Cost-effectiveness of Services for Schizophrenia in the UK across the Entire Care Pathway in a Single Whole-Disease Model. JAMA Network Open. 2020;3(5):205888. doi: https://doi.org/https://doi.org/10.1001/jamanetworkopen.2020.5888.
- 32. Lord J, Willis S, Eatock J, Tappenden P, Trapero-Bertran M, Miners A, et al. Economic modelling of diagnostic and treatment pathways in National Institute for Health and Care Excellence clinical guidelines: the Modelling Algorithm Pathways in Guidelines (MAPGuide) project. Health Technology Assessment. 2013;17(58):v-vi, 1–192. Epub 2013/12/12. pmid:24325843; PubMed Central PMCID: PMC4781470.
- 33. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339. pmid:19622551
- 34. Drummond M F., Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. British Medical Journal. 1996;313:275–83.
- 35. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS)—explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value in Health. 2013;16(2):231–50. Epub 2013/03/30. pmid:23538175.
- 36. Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technology Assessment. 2004;8(36):iii–61. pmid:15361314
- 37.
National Institute for Health and Care Excellence. Developing NICE guidelines: the manual. London, UK: National Institute for Health and Care Excellence; 2014.
- 38. Lauer JA, Röhrich K, Wirth H, Charette C, Gribble S, Murray CJL. PopMod: a longitudinal population model with two interacting disease states. Cost Effectiveness and Resource Allocation. 2003;1(1):6. pmid:12773215
- 39. Ginsberg GM, Lauer JA, Zelle S, Baeten S, Baltussen R. Cost effectiveness of strategies to combat breast, cervical, and colorectal cancer in sub-Saharan Africa and South East Asia: mathematical modelling study. BMJ (Clinical research ed). 2012;344:e614. pmid:22389347
- 40. Stanciole AE, Ortegón M, Chisholm D, Lauer JA. Cost effectiveness of strategies to combat chronic obstructive pulmonary disease and asthma in sub-Saharan Africa and South East Asia: mathematical modelling study. BMJ. 2012;344:e608. pmid:22389338
- 41. Ortegon M, Lim S, Chisholm D, Mendis S. Cost effectiveness of strategies to combat cardiovascular disease, diabetes, and tobacco use in sub-Saharan Africa and South East Asia: mathematical modelling study. BMJ (Clinical research ed). 2012;344:e607. pmid:22389337
- 42. Salomon JA, Carvalho N, Gutiérrez-Delgado C, Orozco R, Mancuso A, Hogan DR, et al. Intervention strategies to reduce the burden of non-communicable diseases in Mexico: cost effectiveness analysis. BMJ. 2012;344:e355. pmid:22389335
- 43. Ginsberg GM, Edejer TT-T, Lauer JA, Sepulveda C. Screening, prevention and treatment of cervical cancer—a global and regional generalized cost-effectiveness analysis. Vaccine. 2009;27(43):6060–79. pmid:19647813
- 44. Ginsberg GM, Lim SS, Lauer JA, Johns BP, Sepulveda CR. Prevention, screening and treatment of colorectal cancer: A global and regional generalized cost effectiveness analysis. Cost Effectiveness and Resource Allocation. 2010;8:2. pmid:20236531
- 45. Stover J, Bollinger L, Izazola JA, Loures L, DeLay P, Ghys PD, et al. What Is Required to End the AIDS Epidemic as a Public Health Threat by 2030? The Cost and Impact of the Fast-Track Approach. PLOS ONE. 2016;11(5):e0154893. pmid:27159260
- 46. Edossa DG, Wedajo AG, Koya PR. Optimal Combinations of Control Strategies and Cost-Effectiveness Analysis of Dynamics of Endemic Malaria Transmission Model. Computational and Mathematical Methods in Medicine. 2023;2023:7677951. PubMed PMID: 2025143452. pmid:37284173
- 47. Coates MM, Sliwa K, Watkins DA, Zuhlke L, Perel P, Berteletti F, et al. An investment case for the prevention and management of rheumatic heart disease in the African Union 2021–30: a modelling study. The Lancet Global health. 2021;9(7):e957–e66. PubMed PMID: rayyan-335941316. pmid:33984296
- 48. Seidu B, Makinde OD, Bornaa CS. Mathematical Analysis of an Industrial HIV/AIDS Model that Incorporates Carefree Attitude Towards Sex. Acta biotheoretica. 2021;69(3):257–76. PubMed PMID: rayyan-335941443. pmid:33502640
- 49.
Tappenden P. A methodological framework for developing Whole Disease Models to inform resource allocation decisions: An application in colorectal cancer. Sheffield, UK: The University of Sheffield; 2011.
- 50. Juusola JL, Brandeau ML. HIV Treatment and Prevention: A Simple Model to Determine Optimal Investment. Medical decision making : an international journal of the Society for Medical Decision Making. 2016;36(3):391–409. pmid:26369347
- 51. Minnery M, Mathabela N, Shubber Z, Mabuza K, Gorgens M, Cheikh N, et al. Opportunities for improved HIV prevention and treatment through budget optimization in Eswatini. PloS one. 2020;15(7):e0235664. PubMed PMID: rayyan-335941225. pmid:32701968
- 52. Kansal AR, Tafazzoli A, Ishak KJ, Krotneva S. Alzheimer’s disease Archimedes condition-event simulator: Development and validation. Alzheimer’s and Dementia: Translational Research and Clinical Interventions. 2018;4:76–88. pmid:29687076
- 53. Zhou H, Isaman DJM, Messinger S, Brown MB, Klein R, Brandle M, et al. A computer simulation model of diabetes progression, quality of life, and cost. Diabetes Care. 2005;28(12):2856–63. pmid:16306545
- 54. Lokkerbol J, Wijnen B, Ruhe HG, Spijker J, Morad A, Schoevers R, et al. Design of a health-economic Markov model to assess cost-effectiveness and budget impact of the prevention and treatment of depressive disorder. Expert review of pharmacoeconomics & outcomes research. 2021;21(5):1031–42. PubMed PMID: rayyan-335941394. pmid:33119427
- 55. Stelmach R, Kocher EL, Kataria I, Jackson-Morris AM, Saxena S, Nugent R. The global return on investment from preventing and treating adolescent mental disorders and suicide: a modelling study. BMJ global health. 2022;7(6). PubMed PMID: rayyan-335941643. pmid:35705224
- 56. Brandeau ML, Lee HL, Owens DK, Sox CH, Wachter RM. A Policy Model of Human Immunodeficiency Virus Screening and Intervention. INFORMS Journal on Applied Analytics. 1991;21(3):5–25.
- 57. Long EF, Brandeau ML, Owens DK. The cost-effectiveness and population outcomes of expanded HIV screening and antiretroviral treatment in the United States. Annals of internal medicine. 2010;153(12):778–89. Epub 2010/12/22. pmid:21173412; PubMed Central PMCID: PMC3173812.
- 58. Ye W, Brandle M, Brown MB, Herman WH. The Michigan Model for Coronary Heart Disease in Type 2 Diabetes: Development and Validation. Diabetes Technology and Therapeutics. 2015;17(10):701–11. pmid:26222704
- 59. Hoogendoorn M, Rutten-Van Molken MPMH, Hoogenveen RT, Al MJ, Feenstra TL. Developing and applying a stochastic dynamic population model for chronic obstructive pulmonary disease. Value in Health. 2011;14(8):1039–47. pmid:22152172
- 60. Basu S, Bendavid E, Sood N. Health and Economic Implications of National Treatment Coverage for Cardiovascular Disease in India: Cost-Effectiveness Analysis. Circulation: Cardiovascular Quality and Outcomes. 2015;8(6):541–51. doi: https://doi.org/https://doi.org/10.1161/CIRCOUTCOMES.115.001994.
- 61. Pandya A, Sy S, Cho S, Alam S, Weinstein MC, Gaziano TA. Validation of a Cardiovascular Disease Policy Microsimulation Model Using Both Survival and Receiver Operating Characteristic Curves. Medical decision making : an international journal of the Society for Medical Decision Making. 2017;37(7):802–14. pmid:28490271
- 62. Booth N, Jula A, Aronen P, Kaila M, Klaukka T, Kukkonen-Harjula K, et al. Cost-effectiveness analysis of guidelines for antihypertensive care in Finland. BMC Health Services Research. 2007;7:172. pmid:17958883
- 63. Goodman CA, Coleman PG, Mills AJ. Cost-effectiveness of malaria control in sub-Saharan Africa. Lancet. 1999;354(9176):378–85. pmid:10437867
- 64. Youn JH, Stevenson MD, Thokala P, Payne K, Goddard M. Modeling the Economic Impact of Interventions for Older Populations with Multimorbidity: A Method of Linking Multiple Single-Disease Models. Medical decision making : an international journal of the Society for Medical Decision Making. 2019;39(7):842–56. pmid:31431188
- 65. Cromwell I. Development and application of a whole disease model of oral cancer to inform health technology management [Text]2019.
- 66. Hiligsmann M, Ethgen O, Bruyere O, Richy F, Gathon H-J, Reginster J-Y. Development and validation of a Markov microsimulation model for the economic evaluation of treatments in osteoporosis. Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research. 2009;12(5):687–96. pmid:19508659
- 67. Hu D, Bertozzi SM, Gakidou E, Sweet S, Goldie SJ. The costs, benefits, and cost-effectiveness of interventions to reduce maternal morbidity and mortality in Mexico. PloS one. 2007;2(1):e750. pmid:17710149
- 68. Wijnen BFM, Thielen FW, Konings S, Feenstra T, Van Der Gaag M, Veling W, et al. Designing and Testing of a Health-Economic Markov Model for Prevention and Treatment of Early Psychosis. Expert Review of Pharmacoeconomics and Outcomes Research. 2020;20(3):269–79. pmid:31195900
- 69. Van Der Heijden AAWA, Feenstra TL, Hoogenveen RT, Niessen LW, de Bruijne MC, Dekker JM, et al. Policy evaluation in diabetes prevention and treatment using a population-based macro simulation model: The MICADO model. Diabetic Medicine. 2015;32(12):1580–7. pmid:26010494
- 70. Watkins D, Lubinga SJ, Mayosi B, Babigumira JB. A Cost-Effectiveness Tool to Guide the Prioritization of Interventions for Rheumatic Fever and Rheumatic Heart Disease Control in African Nations. PLoS Neglected Tropical Diseases. 2016;10(8):e0004860. pmid:27512994
- 71. Mihalopoulos C, Cadilhac DA, Moodie ML, Dewey HM, Thrift AG, Donnan GA, et al. Development and application of Model of Resource Utilization, Costs, and Outcomes for Stroke (MORUCOS): An Australian economic model for stroke. International Journal of Technology Assessment in Health Care. 2005;21(4):499–505. pmid:16262974
- 72. Sluijs T, Lokkers L, Ozsezen S, Veldhuis GA, Wortelboer HM. An Innovative Approach for Decision-Making on Designing Lifestyle Programs to Reduce Type 2 Diabetes on Dutch Population Level Using Dynamic Simulations. Frontiers in public health. 2021;9:652694. PubMed PMID: rayyan-335941457. pmid:33996729
- 73. Baltussen R, Smith A. Cost effectiveness of strategies to combat vision and hearing loss in sub-Saharan Africa and South East Asia: mathematical modelling study. Bmj. 2012;344:e615. Epub 2012/03/06. pmid:22389341; PubMed Central PMCID: PMC3292524 at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work.
- 74. Chisholm D, Naci H, Hyder AA, Tran NT, Peden M. Cost effectiveness of strategies to combat road traffic injuries in sub-Saharan Africa and South East Asia: mathematical modelling study. BMJ. 2012;344:e612. pmid:22389340
- 75. Chisholm D, Saxena S. Cost effectiveness of strategies to combat neuropsychiatric conditions in sub-Saharan Africa and South East Asia: mathematical modelling study. BMJ. 2012;344:e609. pmid:22389339
- 76. Slawomirski L, Auraaen A, Klazinga NS. The economics of patient safety. 2017. doi: doi:https://doi.org/https://doi.org/10.1787/5a9858cd-en.
- 77. Khan MA, Soteriades ES, King J, Govender R, Hashim MJ, Masood-Husain S, et al. Global Trends and Forecast of the Burden of Adverse Effects of Medical Treatment: Epidemiological Analysis Based on the Global Burden of Disease Study. Cureus. 2020;12(3):e7250-e. pmid:32195068.
- 78. Cooper N, Coyle D, Abrams K, Mugford M, Sutton A. Use of evidence in decision models: An appraisal of health technology assessments in the UK since 1997. Journal of Health Services Research and Policy. 2005;10(4):245–50. pmid:16259692
- 79. Sullivan SD, Mauskopf JA, Augustovski F, Jaime Caro J, Lee KM, Minchin M, et al. Budget Impact Analysis—Principles of Good Practice: Report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force. Value in Health. 2014;17(1):5–14. pmid:24438712
- 80.
Canadian Agency for Drugs and Technologies in Health. Guidelines for the Economic Evaluation of Health Technologies: Canada. Ottawa, Canada: 2017.
- 81. Briggs AH, Weinstein MC, Fenwick EA, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making. 2012;32(5):722–32. Epub 2012/09/20. pmid:22990087.
- 82. Brennan A, Chick SE, Davies R. A taxonomy of model structures for economic evaluation of health technologies. Health Economics. 2006;(12):1295–310. pmid:16941543
- 83. Roberts M, Russell LB, Paltiel AD, Chambers M, McEwan P, Krahn M. Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-2. Medical Decision Making. 2012;32(5):678–89. Epub 2012/09/20. pmid:22990083.
- 84. Barton P, Bryan S, Robinson S. Modelling in the economic evaluation of health care: Selecting the appropriate approach. Journal of Health Services Research and Policy. 2004:110–8. Epub 2. pmid:15099459
- 85. Jin H, Robinson S, Shang W, Achilla E, Aceituno D, Byford S. Overview and Use of Tools for Selecting Modelling Techniques in Health Economic Studies. Pharmacoeconomics. 2021;39(7):757–70. Epub 2021/05/21. pmid:34013440.
- 86. Stahl JE. Modelling methods for pharmacoeconomics and health technology assessment: an overview and guide. Pharmacoeconomics. 2008;26(2):131–48. Epub 2008/01/18. pmid:18198933.
- 87. Cooper K, Brailsford CS, Davies R. Choice of modelling technique for evaluating health care interventions. Journal of the Operational Research Society. 2007;58(2):168–76.
- 88. Karnon J. Alternative decision modelling techniques for the evaluation of health care technologies: Markov processes versus discrete event simulation: Health Economics. 12 (10) (pp 837–848), 2003. Date of Publication: 01 Oct 2003.; 2003.
- 89.
National Collaborating Centre for Cancer. Colorectal cancer: the diagnosis and management of colorectal cancer (CG131). London, UK: National Institute for Health and Care Excellence; 2011.
- 90.
National Collaborating Centre for Cancer. Prostate cancer diagnosis and treatment. NICE clinical gudeline (CG58). London, UK: National Institute for Health and Care Excellence; 2008.
- 91.
National Clinical Guideline Centre. Atrial fibrillation: the management of atrial fibrillation: NICE clinical guideline (CG36). London, UK: National Institute for Health and Care Excellence; 2006.
- 92. Williams A. What Could be Nicer than NICE? London, UK: Office of Health Economics; 2004 20 March 2018.
- 93. Monks T, Currie CSM, Onggo BS, Robinson S, Kunc M, Taylor SJE. Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines. Journal of Simulation. 2019;13(1):55–67.