Introduction

Decision analysis is a systematic approach to decision making under uncertainty [1]. The application of decision-analytic models (i.e. simulation models) allows the evaluation of new technologies (e.g., treatments, vaccinations, interventions against the spread of the disease) with respect to benefits, risks, harms and costs. Such computer simulation models are necessary where randomized controlled clinical trials or observational studies are missing, or not feasible [2]. Decision-analytic models (DAM) synthesize available information from various sources (e.g., epidemiological studies on the natural history of the disease or the spread of the disease, short-term clinical studies on treatment effects, quality of life studies, expert opinions) for their short- and long-term evaluations [3].

For the evaluation of interventions in the area of infectious diseases, the transmissible nature of infectious diseases plays a central role when developing a model. The reduction of the prevalence of infections in the target population often leads to a decreased risk for other individuals in the community. As a best case, the infectious disease may even be eliminated if the number of infected individuals decreases below a critical number. It is therefore paramount for decision-analytic models to capture not only the direct effect of interventions on the individuals under intervention but also indirect effects on the population level including herd immunity [4–7]. Herd immunity refers to the effect that a partly vaccinated population can prevent the spread of an infectious disease, and thereby protect unvaccinated persons. In addition, the possibility of acquiring natural immunity following recovery from infection should be considered.

The recent guidelines of the ISPOR-SMDM Modeling Good Research Practices Task Force [4, 8, 9] note that some modeling methods for decision-analytic models in communicable diseases, such as Markov state-transition cohort models, ignore the indirect effects that arise from averted incident infections, and therefore benefits and cost-savings from the intervention may be underestimated [2, 7]. In the review of Kim et al. [10], the authors summarize that model choice for cost-effectiveness analyses for vaccination programs need to be improved, for instance, to capture the effect of herd immunity. Jit and Brisson [5] state that many published decision-analytic modeling studies in infectious diseases do not take into account the specific features of infectious diseases such as the transmissibility from infected to susceptible individuals and the uncertainties arising from complex natural history and epidemiology.

In our article, we will illustrate methodological issues in disease transmission modeling using dengue fever as an exemplary infectious disease. Dengue is a mosquito-borne tropical infectious disease caused by the dengue virus. The dengue virus is transmitted by several species of mosquitoes thriving in tropical countries, principally by Aedes aegypti [11, 12]. Local and regional dengue outbreaks are observed in many tropical countries, and increased number of cases is observed during rainy season [13]. Dengue resurged in the 20th century and received public awareness in the past few decades when epidemics became stronger and more severe [14]. With now half of the world’s population being at risk, dengue epidemics are a severe public health problem in tropical countries, causing a large and continuing morbidity and mortality burden. The Global Burden of Disease Study 2013 estimated a total of 576,900 (330,000–701,200) years of life lost to premature mortality attributable to dengue in 2013 and dengue was responsible for 1.14 million disability-adjusted life-years in 2013 [15, 16].

To model dengue epidemics, several modeling approaches have been applied. For example, differential equations models are used to predict the outcome of vaccinations [17] or to analyze epidemiologic aspects [18, 19]. However, differential equations models are inflexible because incorporation of details would require a high (and potentially unmanageable) number of disease compartments. Sitepu et al [20] predicted dengue epidemics using a statistical autoregressive model. However, such statistical methods are limited to specific scenarios and do not incorporate dynamic effects. Several compartment and stochastic simulation models have been applied to predict the impact and cost-effectiveness of Dengvaxia, the first available dengue vaccine [21]. Hladish et al presents two studies of agent-based models of humans and mosquitoes for dengue transmission [22, 23]. A recent review of health economic evaluation studies of dengue vaccines found that out of 13 studies, 8 studies used dynamic mathematical models of dengue transmission, and 2 studies used a Markov model [24].

For evaluations of the effect of interventions on disease spread, the international guidelines developed by the ISPOR-SMDM Modeling Good Research Practices Task Force recommend modeling methods that capture interaction, such as dynamic transmission models [25]. Dynamic transmission modeling is often performed using system dynamics. System dynamics models are deterministic and based on compartments. However, these deterministic compartment models cannot capture stochastic effects such as the extinction of the disease in small populations and complex interactions and behavior. Agent-based models (ABM) are a more flexible alternative approach for dynamic transmission modeling because AMBs are modeling individuals rather than compartments [4].

The aim of our study is, therefore, to discuss the concepts and important characteristics of ABM for decision-analytic modeling of infectious diseases. In a case study of dengue epidemics, we apply an ABM model to analyze the impact of potential efficacy of mosquito control interventions on the number of new dengue infections in humans in the Philippines. The case study serves to illustrate model characteristics, conceptualizing the model, experiences with model calibration and model analysis.

Case example dengue

Dengue is a mosquito-borne tropical disease that can result in a fever with a mild to life-threatening progression [26–28]. There are five serotypes of dengue virus [29]. After an infection, a person is immune against the specific serotype. However, studies show that further infections with other serotypes often result in more severe disease progressions [12, 30].

In the Philippines, dengue is prevalent for the entire year with a small number of infections during dry season and significantly higher infection numbers during the rainy season [13]. The prevalence of dengue infections in the Philippines also differs among regions. However, the underlying mechanisms of dengue epidemics are not yet completely understood. Why an outbreak happened at a certain place and time is hard to explain.

Measures against the spread of dengue can be categorized into three types of interventions: (1) prevention from being bitten by mosquitoes (e.g., by repellents), (2) vector control, such as releasing Wolbachia infected or genetically modified mosquitoes, sterilizing techniques, and mass-trapping leading to a reduced number of mosquitoes [31], and (3) vaccinations. The first two measures have been the default interventions for many years. To date, vaccines have been developed and tested in long clinical trials and are now publicly available in selected countries since early 2016 [13, 32, 33].

Within our project, we aimed building an agent-based model which is suitable to simulate a dengue outbreak in a Philippine city or region. The model allows analyzing the mechanisms of dengue epidemics and evaluating the impact of mosquito (vector) control interventions on disease burden. In this paper, we evaluated the impact of human mosquito ratios on disease incidence to show the effectiveness that a future mosquito control should reach to stop the epidemics. Before we describe the model in detail, we summarize and discuss characteristics of ABM.

Characteristics of ABM and reasons for model selection

Agent-based modeling focuses on complex systems composed of ‘agents’ that are uniquely identifiable, autonomously acting, and capable of interactions [34–36]. Agents are often human individuals but can also be animals, or objects such as vehicles, buildings etc.. These agents can capture population heterogeneity. The agents’ behaviors are often described by simple rules, for example, contacts within social networks or contacts with mosquitos. Agents’ behaviors in combination with transmission pattern and disease progression result in emerging population-wide dynamics such as the outbreak of a disease [37]. In the following, the main characteristics of ABM are discussed with the focus on infectious diseases and the case example of a dengue fever epidemic.

Applied ABM for dengue

In this section, we describe the framework, structure, parameters, calibration, validation of the ABM we developed for the evaluation of dengue interventions to control dengue, and the analytic techniques of the evaluation. A detailed model description is provided in the supplementary material S5 File. The model has been implemented in Java SE 8.The program code will be provided upon request.

Model parameter and data

To simulate the dengue epidemic in Cebu City, hospital data containing all hospitalized dengue cases in 2010 were applied. The datasets were provided by the City Health office in Cebu City for use in this project (S1 File). The human population was defined according to the Cebu City population of 2010, based on data of the Philippine Census [46]. All model parameters are listed in Table 1, the computations are presented in S2 File. Parameters derived by calibration are marked accordingly. The calibration process is described thereafter.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Model parameters of the dengue model and sources.

https://doi.org/10.1371/journal.pone.0221564.t001

The provided datasets about dengue cases were anonymized and did not include any identifiable information. The population data from the Philippine Census is aggregated and does not allow to identify persons. The agents in the model are objects representing aggregated data and do not relate to existing persons.

Calibration

The human population and the reported dengue cases are well known. However, in addition to the reported cases, there are mild unreported cases and asymptomatic cases. These persons are unknown but they are infectious and can transmit the disease. The number of asymptomatic and unreported dengue cases were estimated based on Chastel et.al. [27].

There is no reliable information about the mosquito population. This includes the population size, lifespan, reproduction numbers, and biting habits. There is some information about incubation period of mosquitoes and that they remain infectious for the rest of their lives. The infection probabilities for humans and mosquitoes are also unknown and are subject for calibration.

The model has been calibrated to the 2010 dengue epidemic in Cebu City for eight unknown model parameters (probability of transmission for persons and mosquitoes, mosquitoes per person during rainy and dry season, mosquito death probability per day, initially infected mosquitoes, and start and end of the rainy season). The epidemic is defined as newly infected persons per week over time based on the hospital case data and assumptions about asymptomatic and unreported cases (Table 1). The calibration goal is to reproduce the incidence numbers.

A manual calibration has been performed because an automated calibration was not possible due to the large number of parameter combinations and time intervals and relatively long simulation times. Calibration results were plotted in order to identify the parameter set that provides the best fit.

The calibration has been performed considering three phases: (1) the situation before the outbreak, (2) the start of the rainy season, and (3) the end of the epidemic. The first calibration phase showed that there is a strong relation between mosquito numbers, biting rates and transmission probabilities. For example, more mosquitoes with less biting rates result in the same human infections. We found some information about biting times of female mosquitoes [47, 48], assumed a uniformly distributed biting time of 5 to 90 seconds and a mosquito population as large as the human population (referred to as “mosquitoes per human” and as a ratio “humans:mosquitoes”), and we assumed that all mosquitoes die at the age of 33 days. The calibration of the first 25 weeks of the year is straight-forward and leads to transmission probabilities for humans and mosquitoes.

In the second calibration phase, first only the number of mosquitoes at week 25 was increased to create an outbreak. Consequently, the epidemic starts as required. The earlier assumed predefined fixed life span of mosquitos of 33 days, however, leads after 33 days to a large number of infectious mosquitoes dying, which are replaced by susceptible newborn mosquitoes. This causes a significant decline in the human epidemic until a significant portion of newborn mosquitoes get infected and pass their incubation period. Therefore, daily dying probabilities and population growth rates were introduced to gain better mosquito demographics. Parameters were adapted until the epidemics between week 25 and 37 were satisfying.

Calibrating the third phase after week 37 was challenging because the developed system is inert and takes weeks to react to changes. Infected mosquitoes live for weeks, and humans are viraemic for several days after almost one week of incubation period. To stop the epidemics after week 37, two attempts were made (Fig 1, S3 File). Attempt 1 tries to reduce the mosquito population as soon as possible at the end of rainy season. Then, there are no newborn mosquitoes for 12 days. After that, the mosquito population has reached its defined number and a regular amount of mosquitoes per day are born. However, there are still a significant number of infected persons due to incubation period and viraemic period. The newborn mosquitoes biting these persons results in an unusually high prevalence among young mosquitoes. Consequently, the epidemic emerges again in week 39 (delayed by the mosquitoes’ incubation period), and a repeated smaller epidemic in week 43 appeared. In Attempt 2 a higher mosquito death rate was assumed, while still new mosquitoes are born. However, this results only in a slow decline because new mosquitoes always become infected quickly by humans.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Calibration attempts.

Fig 1 displays the results of the calibration attempts in comparison to the real-world data. Phase 1 and 2 are similar, using the two mosquito population adaptation techniques, while phase 3 results in different behavior.

https://doi.org/10.1371/journal.pone.0221564.g001

Finally, the model was calibrated following the second attempt of mosquito population adaptation. The best fit can be achieved when the mosquito population during rainy season rises to three times the human population, and declines to the same size as the human population after the rainy season. The resulting epidemic is presented in Table 1, Fig 2 and the supplementary material S4 File.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Timeline of the dengue incidence in humans in the year 2010.

It compares the reported dengue cases to simulation results. 1:1.0, 1:1.5, 1:2.0, 1:2.5, and 1:3.0 refer to simulations with different human:mosquitoes ratios during raining season. The human:mosquito ratio describes the number of mosquitoes depending on the number of humans. This allows us to change the number of humans in the model without the need to change the number of mosquitoes, e.g., for scaling the model. The calibration result refers to the simulation with the calibrated parameters.

https://doi.org/10.1371/journal.pone.0221564.g002

We validated the model according to the ISPOR-SMDM Good Modeling Practice Guidelines on several levels: (1) face validity (i.e., by public health experts, modeling experts), and (2) internal validation (debugging, consistency and plausibility checks) [51]. For internal validation, we also applied a simplified version of immersive face validation [52]. Immersive face validation involves observing a simulation from the perspective of a single agent. This provides different information than from aggregated model results, which typically represent the whole system. With immersive face validation, the behaviors of single agents are tested, ideally using a graphical 3D-representation of the system. Sophisticated forms of this validation even enable the programmer to perform actions with the agent during simulation time. In our dengue model, we used a simplified version of immersive face validation to observe, how often humans are bitten by mosquitoes, how many of them were infected and how often a mosquito bites viraemic and non-viraemic humans.

Results

Fig 2 displays the incidence of infected people estimated by the calibrated model and the real-world data of Cebu City 2010. Predominantly we see a good fit of the curves. The simulated decline of the epidemics is slightly delayed.

The impact of potential mosquito control strategies on the number of new infections per week is displayed in Fig 2, it shows that a reduced human-mosquito ratio of 1:2.5 and 1:3 leads already to a substantial decrease of infected humans and smaller epidemics. Assuming a human mosquito ratio of 1:2 or smaller, the negative trend (i.e., decrease of new infections) during week 1 to 25 continues also during rainy season until the infection numbers reach zero. If a future intervention can reduce the ratio to 1:1 (i.e., number of mosquitoes does not increase during rainy season), the dengue epidemic eventually dies out. (see also S4 File.)

Discussion

ABM is a dynamic modeling approach that is able to reproduce direct and indirect effects of interventions for communicable diseases and to replicate emerging behavior. We summarized major characteristics of ABM and illustrated in a case example how ABM can be applied to model a dengue outbreak and to evaluate the effect of potential mosquito control interventions differing in effectiveness.

Classical approaches for epidemic modeling often use ordinary differential equations or system dynamics, where aggregated variables represent the population of interest that is grouped by health and other attributes [18, 53–55]. Because of this aggregation level, model development, validation and sensitivity analyses tend to be faster as compared to individual-based models. However, the high abstraction level leads to less flexibility for subgroup analyses if this was not specified upfront (e.g., vaccination of all persons of a certain age that are frequently bitten by mosquitoes). In contrast, agent-based modeling simulates individuals. An individual-based simulation allows one to include and track patient characteristics and medical history, evaluation of subgroup specific interventions, calculating specific outcomes and distributions of these outcomes as well as modeling an open cohort. A specific distinguishing feature of ABM compared to other individual-based modeling approaches is that the behavior of agents and respective emerging behavior of the epidemics [56–58] can be captured. Changing the behavior of specific groups of agents is possible with less programming effort. In a compartmental model (e.g., system dynamics) this would require a restructuring of the compartments and respective underlying equations.

The flexibility of agent-based models supports the style of iterative modeling. Starting with a reduced and simplified model structure, more features (e.g. agent characteristics, potential disease pathways) can be added iteratively until the model complexity meets the needs of the respective (future) model applications. Reusability is an important issue because it can significantly reduce the effort for model development, and working with validated model parts increase both validity and credibility in the scientific community and for health policy decision makers. A disadvantage of individual-level ABM are longer conceptualization and model implementation times.

ABMs are constructed to reproduces an epidemic in a similar way as it happens in reality. The spreading process occurs based on agents’ behavior. Hence, the model gives insights into the mechanics of the underlying process, how the epidemic spreads among the population, and how system changes affect this spreading process, which eventually results in different outcomes. In particular, this allows to examine herd immunity and serotype shifts. Stochastic state transition models cannot produce an emerging behavior. They can only incorporate them if they are known for each situation, which is usually not possible for scenarios. In contrast to that, the agent-based model, where the epidemic emerges upon basic rules, automatically simulates these effects without incorporating them explicitly. Yet it is a matter of current research to find a commonly accepted definition and a standardized way for measurement of herd immunity and serotype shift, which makes it even harder to use them in models where they are explicitly required [59].

In ABM, it is technically even possible to trace back the mosquito that infected a person, when and where it was born, got infected, how many persons it infected before, and what it did afterwards until it died. Such information could support model validation and in particular the novel immersive face validation.

Compared to other modeling approaches, ABM also has limitations. ABM usually requires longer development time due to complexity. Because ABM are individual based, one faces longer simulation time, lack of analytical tractability, and challenges in parameterization [60]. For example, often there is a lack of studies on contact pattern or frequencies. Therefore, models require extensive calibration efforts for unknown parameters. However, these unknown parameter are made explicit and therefore they become accessible for discussions. Emergent behavior is both an advantage and a weakness of agent-based models. It allows simulating complex and even counterintuitive dynamics. However, the cause of wrong dynamics as a result of erroneous or incomplete agent’s behavior are hard to detect. Therefore, Klügl [61] presents a validation approach specifically for agent-based models that should help to detect such problems. Marshall et al. discuss a new dimension of sensitivity analyses, that is added and may be challenging in ABM: testing the assumptions about human behavior (e.g., “how people learn, how they disseminate information to their peers or families, and how they change their behavior in response to new information, incentives, or penalties”) in ABM [40]. Chattwahl and He [38] discuss challenges of probabilistic sensitivity analyses due to transmission and network-related parameters that are correlated. In addition, probabilistic sensitivity analysis can require long simulation time because of the combination of first- and second-order uncertainty. Value of information analysis to discover parameters of future research would also be computational very intense. Modeling an open cohort leads to the question of how outcomes are summarized. For example, within a cost-effectiveness analysis of vaccinations of newborns one could consider the outcome (e.g., health-related quality of life) for the entire population over a fixed time horizon or the outcome summarized for people that have at least a follow up time of ten years. Finally, as with all complex models, visualization of ABM with respect to the several modules or layers (development of population, disease, contacts) could be challenging.

Literature on ABM comprises technical tutorial papers with general applications [4, 37], simulating human systems [62], tutorial on economic evaluation using ABM [38], teaching ABM [37] and comparison of modeling approaches and guidance on model selection [4, 40, 41, 63]. In contrast to this existing literature, we provided a tutorial example on why and how to apply ABM for decision analysis in infectious diseases, and on limitations.

In principle, the concept of ABM is independent of any particular software. However, there are specific software packages that have been designed to be applied or to integrate ABM. Here, we name an exemplary selection of available software. Beyond several commercial products such as ANYLOGIC (The AnyLogic Company), there are open-source agent-based simulators such as MASON, which has also been used to simulate epidemics [64] and EpiSimS, an agent-based simulation engine for modeling the spread of disease in regions developed by Los Alamos National Laboratory in the United States [65]. Repast, an open-source toolkit developed at the University of Chicago, is a rather general framework for agent-based modeling that provides some basic functionality while the users have to implement the model themselves in JAVA, .net, or Python [66].

Technical model implementations do usually not follow strict guidelines or instructions. Instead, they require an algorithmic approach that heavily depends on the actual problem. Hence, existing ABM simulators often have shortcomings for specific models because they cannot consider all possible requirements. In these cases, the simulator must be extended by the user or one decides to implement the model from scratch in a general programming language. Thus, technical programming experience with the specific software is crucial for ABM modeling and advanced general computer programming skills are often necessary.

The case example of dengue epidemics is challenging because the disease is not directly transmitted from human to human but from humans to mosquitoes and mosquitoes to humans. This requires modeling of two different populations leading to complex dynamics. The agent-based dengue model illustrates how these complex dynamics can be reproduced to a great extent, but producing the exact emerging behavior would require increased model complexity.

In our modeling example, each day 20% of the mosquitoes die and are replaced by new mosquitoes. This requires deletion and creation of a large number of agents, which affects the computation performance. Thus, the runtime heavily depends on the actual implementation, on used data structures and RAM management. The model has been implemented using JAVA based on a framework developed in earlier projects [42]. A sample simulation of 365 days with the full population on an Intel^® Xeon^® 3.3 GHz processor runs in 9 minutes and 54 seconds and needs a maximum of 600 MB RAM during dry season and 1.3 GM during rainy season.

Our case example has the following major limitations. In the population model, aging, deaths, births, emigration and immigration of the human population are not considered. This assumption can be reasonable for short-term analyses. For future long-term analyses however, open cohort dynamics should be implemented. We did not consider medical history including prior dengue infections with a different serotype that impacts the severity of the disease. Available data to model past dengue infections are limited. With respect to the mosquito incidence, we did not model in detail regionality of breeding places and the growth of the mosquito population during rainy season. The mosquito population size and age distribution, and population dynamics such as growth and decline at the beginning and end of the rainy season have been widely calibrated due to a lack of specific data. The decline of the mosquito population at the end of the rainy season in the calibrated model was slightly slower than the decline displayed by the real world data. This gap is assumed to be caused by unknown biological processes in the mosquito population. For example, the cause of the declining population could be an increase in death rate or a decrease in birth rate. Therefore, the emergent behavior at the end of the rainy season could not be exactly estimated. A common challenge for epidemics also strikes our dengue example: epidemics often occur in yearly, seasonal outbreaks. However, these outbreaks are usually completely different from year to year. Epidemiologists often can only provide limited explanations; therefore the goodness of a model prediction of any upcoming seasonal epidemic is limited. The only sensible approach is to use as an example a single, well-known epidemic from the past to test the effects of interventions. Applying an epidemic model to another season usually requires a fresh calibration.

Finally, we did not include adverse effects (e.g., of repellents, resistance) on health-related quality of life or the system itself and costs for our demonstration purposes but cost-effectiveness analysis could be performed as previously shown [38].

Further interventions that may be considered are: preventing from being bitten by mosquitoes (e.g., by repellents) and vaccinations. Due to the flexibility of our modular ABM, vaccinations can be implemented with manageable effort. Only one additional attribute needs to be added that characterizes an immune person, and a structure that incorporates vaccination strategies. This is an excellent example for flexibility since it does not affect the rest of the model structure.

In our case example, calibration—in particular of the declining epidemics—can be improved. Reasons of the sudden and very steep decline in our case example are not fully understood by epidemiologists. Hence, implementation of the underlying processes is difficult and may require adding additional complexity to the model. In comparison to other studies, for example the Yucatán model published by Hladish, we see a wide range of similarities [22, 23]. In the Yucatán model, the mosquito population starts to grow in May and reaches its maximum in July, before it declines in September. The number of dengue cases starts increasing in July, reaches its maximum in October, and then slowly decreases until January. Our model could reproduce such a slow decrease of dengue cases. However, the dengue epidemic in the Philippines stops abruptly within three weeks. Due to the long delay between size of mosquito population and dengue epidemic, we could not fully reproduce the rapid decline.

Our model could be further improved with regard to regionality, seasonality, serotypes, patient history and adaptive behavior. Incorporating households and the detailed location where persons and mosquitoes live can, for example, support the evaluation of targeted (regional) vaccination strategies and regional measures to contain the mosquito population. Incorporating seasonality (i.e., the impact of rain seasons on mosquito growth) could improve estimated monthly health care demand and subsequently support planning of health care provision. To differentiate between the four dengue serotypes in the model could also improve the evaluation of vaccination strategies. Studies have shown that persons can only become resistant against their serotype of infection, and that they are more likely to develop severe symptoms during secondary infections [12, 30]. Therefore, vaccination against specific serotypes could also prevent severe symptoms of secondary infections which may be considered as an additional benefit in a cost-effectiveness analysis. The model could be refined to incorporate patient histories including previous infections or comorbidities to allow, for example, for predictions on the severity of the disease. Further advanced topics such as adaptive behavior (e.g., humans taking prevention against mosquito bites, mosquitos getting resistant against certain measures) could also be considered in decision analyses.

In general, for the application of ABM for infectious diseases, there are several open questions for further research with respect to model validation and uncertainty analysis. For model validation a novel face validation technique where a human expert checks systematically what an agent perceives and how it reacts (immersive face validation) deserve closer attention [52]. For sensitivity analysis, testing the assumptions about human behavior [41] and doing a probabilistic sensitivity analysis within a reasonable time and taking into account correlation between parameters remain challenging [38].

Conclusions

ABM is a powerful tool to support decision-analyses for infectious diseases including the dynamics of infectious diseases and emerging behavior, and with a high level of reusability due to modularity. Conceptualizing an ABM, the level of required detail should carefully be considered based on research questions to define a reasonable complexity. The ability to include human behavior or the behavior of other agents makes assumptions explicit, accessible for discussion but may also lead to extensive calibrations due to a lack of data. The complexity of ABM and the underlying structure often requires longer modeling, validation and simulation time. Models can become very data intense. Probabilistic sensitivity analysis can be challenging because of required information on parameter correlation and long simulation time because of the combination of first- and second-order uncertainty. Our case example illustrates that intense calibration efforts may lead only to reasonable reproducibility of reality. The focus of future research should be on model calibration and validation.

Supporting information

Acknowledgments

We thank Dr. Lyndon James (Harvard T.H. Chan School of Public Health) for proofreading and language editing.