Figures
Abstract
Agent-based models (ABM) are widely used to study immune systems, providing a procedural and interactive view of the underlying system. The interaction of components and the behavior of individual objects is described procedurally as a function of the internal states and the local interactions, which are often stochastic in nature. Such models typically have complex structures and consist of a large number of modeling parameters. Determining the key modeling parameters which govern the outcomes of the system is very challenging. Sensitivity analysis plays a vital role in quantifying the impact of modeling parameters in massively interacting systems, including large complex ABM. The high computational cost of executing simulations impedes running experiments with exhaustive parameter settings. Existing techniques of analyzing such a complex system typically focus on local sensitivity analysis, i.e. one parameter at a time, or a close “neighborhood” of particular parameter settings. However, such methods are not adequate to measure the uncertainty and sensitivity of parameters accurately because they overlook the global impacts of parameters on the system. In this article, we develop novel experimental design and analysis techniques to perform both global and local sensitivity analysis of large-scale ABMs. The proposed method can efficiently identify the most significant parameters and quantify their contributions to outcomes of the system. We demonstrate the proposed methodology for ENteric Immune SImulator (ENISI), a large-scale ABM environment, using a computational model of immune responses to Helicobacter pylori colonization of the gastric mucosa.
Citation: Alam M, Deng X, Philipson C, Bassaganya-Riera J, Bisset K, Carbo A, et al. (2015) Sensitivity Analysis of an ENteric Immunity SImulator (ENISI)-Based Model of Immune Responses to Helicobacter pylori Infection. PLoS ONE 10(9): e0136139. https://doi.org/10.1371/journal.pone.0136139
Editor: Filippo Castiglione, National Research Council of Italy (CNR), ITALY
Received: March 7, 2015; Accepted: July 31, 2015; Published: September 1, 2015
Copyright: © 2015 Alam et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: The data can be obtained without restriction after creating an account from: https://immport.niaid.nih.gov/immportWeb/clinical/study/displayStudyDetails.do?itemList=SDY230. The Study number is SDY230.
Funding: This work has been partially supported by Defense Threat Reduction Agency (DTRA)—R&D (HPC) Award No. HDTRA1-09-1-0017, DTRA—Validation Award No. HDTRA1-11-1-0016, DTRA—Comprehensive National Incident Management System (CNIMS) Award No. HDRTA1-11-D-0016-0001, National Science Foundation (NSF) PetaApps Grant OCI-0904844, NSF Network Science and Engineering (NetSE) Grant CNS-1011769, NSF Software Development for Cyberinfrastructure (SDCI) Grant OCI-1032677, National Institutes of Health (NIH) MIDAS project 2U01GM070694-7, and National Institute of Allergy and Infectious Diseases (NIAID) & NIH project HHSN272201000056C.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Sensitivity analysis (SA) is the study of the impact of different parameters in the outcomes of a system [1, 2]. SA has been used in many scientific fields to highlight important data, optimize the design of a system, simplify models, and rank the influence of various parameters on a given system [3–6]. SA can be performed in a local or global context. Local SA examines the effect of deviations of a parameter (within its range), on system outcomes around a base setting [7], i.e., only one parameter is changed while all others remain fixed. Global SA evaluates the entire parameter space to determine all of the system’s critical points [3, 8]. Both statistical and deterministic methods can be used for SA [2, 9]. Other approaches for SA include variance decomposition [10], response surface [11, 12], and sampling methodologies [13]. The choice of method depends on the purpose and the system under study. Typically the systems under review are computationally expensive and hence SA becomes very challenging.
ABM is a computational technique where the behavior of individual agents or groups of agents is encoded by simple rules, and the outcomes are observed at the system scale. ABM is a widely used technique in different areas of study such as computer science [14], economics, biology, ecology, and social phenomena. Popularity of ABM is also due to its intrinsically spatial component [15] and the fact that the system allows integration of different experimental data into one in silico experimental system very easily [16]. In fact, this modeling framework can be used to intuitively map biological (or social) phenomena and at the same time capture some of the inherent stochastic nature of these systems without an added layer of complexity [15, 17–20]. Furthermore, ABM offers modeling intuitive implementation of interactions (e.g. binding of molecules) where rules that describe the interactions are often taken from existing literature. A concrete example: a large body of studies have successfully used ABM to simulate dynamics of inflammation [21–26]. ABM has become even more popular during the past decade with the exponential growth of technological and computational power. For instance, in 2008, agent based modeling and simulation studies were used to explore the development and resolution of virtual infections in a unique way [27, 28]; the study was fundamental in the identification of key parameters capable of inducing clearance, persistent infection, or even death. For a comprehensive review of ABM applied to host-pathogen systems, the reader is referred to a review by Bauer et al., [29], where authors describe various models but also discuss some of the limitations and challenges.
We use a formal mathematical framework called graphical dynamical systems (GDS) [30, 31] (see S1 Text for definition) to represent the agent based models studied here. This representation elucidates the formal aspects of the model as a discrete and finite dynamical system. GDS is a widely used framework for capturing distributed, dynamical systems such as social contact graphs [32], urban traffic and transportation [33], epidemics [30], dynamics of biological systems [34], and gene annotation [35]. The configuration space of an ABM, represented as a GDS G, can be conveniently viewed as a discrete Markov chain M. Each node in M is the state vector of the node states in G. A directed edge from node A to B in M with label p denotes the probability of transition from A to B. The nodes of M are also referred to as the state space of M. For example, a complex ABM comprised of one million agents, each of these agents having six possible states, results in a Markov chain with a state space consisting of 6106 different configurations. Note that the Markov chain is represented succinctly by the ABM. The chain is exponentially larger than its representation. Straightforward methods for global sensitivity analysis on such large chains are unlikely to scale. It is worth pointing out that the proposed sensitivity analysis are not restricted to GDS. It is rather a general methodology applicable to a wide range of complex simulation models. However, we believe having a good understanding of the presented system can be beneficial as it can provide a direct example of a complex system where such a method can be used. The size of the state space makes the experimental design of interest. If the state space was small, one could instead use many other designs that can exhaustively cover the range of parameter values.
In this paper, we perform sensitivity analysis on a complex ABM named ENteric Immunity SImulator (ENISI) [36–38]. ENISI is a modeling environment for studying the inflammatory and regulatory immune responses in the gut. During the development of ENISI, three versions were released; ENISI HPC [39], ENISI Visual [40], and ENISI MSM [41, 42]. ENISI HPC focuses on scalability by implementing a parallel simulation framework. ENISI Visual focuses on visualizations. ENISI MSM, a multiscale modeling framework, focuses on the integration of heterogeneous modeling technologies. The SA method presented here can therefore be utilized directly for the three different technologies in addition to being valuable to other ABM systems. The novelty lies in the choice of appropriate designs that enable statistical methods to explore parts of the Markov chain in a computationally efficient manner and still guarantee certain statistical properties. We use orthogonal arrays and variation-based sensitivity analysis to achieve this, which is discussed in detail in subsequent sections.
Developing novel approaches for sensitivity analysis of agent-based models enhances understanding of the influence of different input parameters and their variations on model outcomes and has the potential to improve the parameter estimation process by making it more systematic and streamlined. The objective of the sensitivity analysis is to identify the greatest and least significant parameters in the model, and to quantify how parameter uncertainty influences outcomes. Analytically exploring the behavior of such systems, even if model outcomes mirror reality, is challenging due to the large number of parameters [2]. Sensitivity analysis is also important for understanding the relationships between input and output variables, for testing the robustness of the system, and for identifying errors in the model.
ENISI: an agent-based modeling environment
ENteric Immunity SImulator (ENISI) is a modeling environment for studying inflammatory/effector and regulatory immune responses in the gut [36–38]. ENISI encodes immune pathways as an agent-based model representing individual cells that participate and interact with other cells in the gastrointestinal tract. The individual cells are distributed spatially to four locations: lumen, epithelial barrier (EB), lamina propria (LP), and gastric lymph node (GLN). Locations are further divided into many discrete patches called sublocations. A sublocation is defined as the maximum volume at which a cell can be assumed to be in contact with all other cells. Sublocations represent the discrete region where cells remain and interact. Each cell moves between sublocations according to its own schedule. A cell can have its own movement schedule assigned by phenotype or state and location. Within a location, cells move between the scheduled sublocations at short time intervals, which represents random movement and produces a dynamic contact network. This random movement, which does not favor movement to adjacent sublocations, is added because of an assumption that cells are moving at a much faster rate than the time period for sublocation update (30 minutes). Therefore, the probability of an individual cell having traveled far from the current sublocation is the same as the probability of being in a nearby sublocation after the 30 minute interval.
Parameters
The ENISI model of immune responses to H. pylori has a set of 25 independent modeling parameters. These modeling parameters include transition probabilities, constant factors of group-cell interactions, stimulation/inhibition factors, and migration rates. Each of the 25 parameters is a continuous input variable within corresponding parameter ranges. Parameter values are obtained from the literature whenever available or estimated during model development [43–48]. A complete list of parameters along with a short description, range, and default value is shown in Table 1.
System output
The ENISI model under study has eight types of immune cells and bacteria (epithelial cells, tolerogenic bacteria, commensal bacteria, dendritic cells, sampling dendritic cells, macrophages, CD4+ T cells and natural T-regulatory cells) involved in the gastric immune responses to H. pylori. Each individual cell has many different functional states called phenotypes. There are a total of thirty different phenotypes in this model. Among those, activated/effector CD4+ T cells and macrophages are the most interesting cell types for the disease model under study. Furthermore, there are four different tissue sites modeled in ENISI: lumen, epithelial barrier (EB), lamina propria (LP), and gastric lymph node (GLN). ENISI provides the number of cells of each phenotype grouped by location as the system output. A complete list of cell types is available in [36].
Aims and challenges
The aims of this study are to (1) evaluate the influence of parameters on model outcomes, (2) rank the parameters based on their global influence, and (3) refine the ABM based on the result of the SA of the parameters. In this study, we used ENISI to recreate an experimental infection of an individual mouse with Helicobacter pylori, a bacteria that colonizes the human stomach chronically causing disease in 15% of carriers. The particular strain in this experiment is the Cag Pathogenicity Island (CagPAI)+ H. pylori strain 26695 [36–38]. However, it is important to indicate that this analysis can be applied to an array of complex biological networks including the model of CD4+ T cell differentiation [49–51], network dynamics of T helper 17 induction and differentiation [52], and network of interactions between mucosal immunity and the gut microbiome during Clostridium difficile infection [53].
The key challenges of the study lie in the size and number of parameters of the model and the computational cost associated with the stochastic nature of the process. A simulation in ENISI consists of a magnitude of 107 individual agents. There are thirty different types of agents and a few hundred individual rules. On a modern high performance computing cluster of 48 nodes, a single run of the simulation takes about ninety minutes on average. Because of this limitation, the runtime of multiple simulations must be taken into consideration. Each of the 25 parameters is considered to be continuous; however, if each of the 25 parameters takes only four different values, then a full factorial experimental design would require 425 runs, which is not possible in terms of the system’s runtime. To alleviate this problem we propose a novel experimental design with an economic number of simulations.
Materials and Methods
Related Works
Because sensitivity analysis can be an effective tool for systematically identifying the most and the least significant parameters, it can provide valuable information on the robustness of biological systems. Summer et al., [54, 55] implemented a SA method by exploring a group-based procedure, where SA is applied to groups of parameters. The result of this analysis can be used to identify the components of the model that are most important in determining the model behavior. This method was applied to a composite model of blood glucose homeostasis that combines models of processes at the sub-cellular, cellular and organ level to describe the physiological system [54–56]. Furthermore, using a simple ABM to assess different methodologies for SA, van Voorn et al. [57] highlighted the importance of using a local as well as global method; while the local sensitivity can capture tipping points, the global approach can provide a summary measure of parameters. Along the same line of research, Schouten et al., [58] compared two methods of SA–local and global and concluded that a mixed approach of SA can lead to a better understanding of model behavior. In the global SA, the authors used an approach based on Monte Carlo sampling of parameters. In a recent study, Ligmann-Zielinska et al. [59] argue that SA as well as uncertainty analysis (UA) are important elements in quantification of model output variability and its sensitivity to inputs. On the basis of this argument, authors utilized variance decomposition of the ABM output to demonstrate how an integrated quantitative UA in combination with SA can be used in ABM development. Related analyses include the work done by Parry et al., [60] using a Bayesian approach for SA for the identification of sensitive elements, and work done by Fonoberova et al., [61] using a variance-based as well as derivative-based global SA in the identification of key elements for model reduction. In this paper we consider both global and local sensitivity analysis of ABM using ANOVA and discuss the most and the least significant parameters which could be used for model reduction as in [61]. Though there are numerous instances of SA applied to ABMs, few attempt to classify the behavior of a parameter. We used main effect plots to classify the behavior of input parameters into four classes in a novel fashion that provides both visual representations of and statistical insights into a parameter. Our choice of four levels reduces the computational costs associated with large scale ABM while regaining valuable insights into the parameter. These statistical analyses along with sparse orthogonal array based experimental design provide a concrete framework to perform SA on any agent based models.
Global sensitivity analysis
In global sensitivity analysis we study the behavior of input parameters on the variation of the model output. Not every modeling parameter has equal effects on system outcomes. Some parameters play more significant roles than others. The goal is to quantify the significance of the parameters based on some measurement.
Global versus Local sensitivity analysis
A local sensitivity analysis addresses sensitivity relative to change of a single parameter value, while a global analysis examines sensitivity with regard to the entire parameter distribution.
Global sensitivity analysis focuses on the variance of model outputs and determines how input parameters influence the output parameters. It is a central tool in sensitivity analysis since it provides a quantitative and rigorous overview of how different inputs influence the output.
Local sensitivity analysis focuses more on a single input’s behavior while other parts remain the same. It is narrow in this aspect as the effect of input parameter is not measured for settings other than the base. Global sensitivity analysis is often preferred when possible, due to its greater detail, but for a large system it is time inefficient to run a model. Local sensitivity analysis method can be preferred because it requires less computation.
Choosing suitable model output.
To measure the sensitivity of the model with respect to modeling parameters we need to select suitable output variables. One challenge regarding a complex system like ENISI is that it has many output variables. One of the outputs provided by ENISI is the number of cells of each cell type in every location. There are about 30 total possible phenotypes for the eight types of cells modeled in ENISI. Measuring sensitivity against all the model outputs is difficult and time consuming. In order to conduct the analysis more effectively, we chose nine output variables that are the primary concerns of immunologists [62, 63]. In this study, three phenotypes of T cells are considered: Th1, Th17, and iTreg in the locations LP and GLN as the primary output of the system. Three types of macrophages: M1 (classically activated, prone to promote inflammation), M2 (alternatively activated with regulatory and pro-resolutory functions) and M0 (precursor of M1 and M2 macrophages) in the LP location are also considered. A list of these nine outputs is provided in Table 2.
ENISI models the immune response and pathology at the gastric mucosa during H. pylori infection. In the stomach H. pylori is mainly found in the mucus layer lining the epithelial cells, and a small fraction is present in the lamina propria (LP). Cells participating in the immune response to the bacteria, such as CD4+ T cells and macrophages, are found in the lamina propria. Cells in the gastric lymph nodes (GLN) region also play an important role in the induction of the immune response. Different phenotypes of T cells and macrophages play vital roles in the inflammation and immuno-regulation process [62–64]. For this reason we chose regulatory T cells, T helper 1, and T helper 17 cells in both LP and GLN sites and M0, M1, and M2 macrophages in the LP site as the output variables. We also have biological lab data available for these cell types for model calibration.
Choosing input parameter values.
In this study, the value of each parameter has continuous values with a fixed range as shown in Table 1. In order to conduct variation-based sensitivity analysis [1], a common approach is to choose several levels for each parameter such that the analysis of variance can be performed in a rigorous fashion. In this work, we consider each parameter to have four discrete values (levels) in increasing order: Level 1 to Level 4. The exact value for each level is dependent on the corresponding parameter range from Table 1. Exact values for the four levels of each parameter are given in S1 File. We observed that most of the parameter values in the default calibrated settings tend to take values lower in the range. This is reflected in our choice of values for the levels. With four levels in each parameter, each input parameter has three degrees of freedom when conducting variation-based analysis. In a simple word, one can fit a polynomial function for each parameter with degrees three. Thus, one can study the significance of the linear, quadratic, and cubic effects of the input parameter on the output. It provides flexibility on investigating how the input parameter affects the output for the agent-based simulation model. If one only chooses two levels for each parameter, then only the linear effect of each parameter could be studied [65]. It is worth noting that for a different agent based model, it may not be feasible to have the same number of levels for each parameter. In this situation, we would consider a mixed-level experimental design [66] to address this challenge. We also like to remark that the choice of four levels provides enough information to extract linear and non-linear effects in this study. However, when having fewer parameters and high computational power, modelers can further increase the number of levels to enrich the predictive power of the analysis.
Design of Experiments
The results of global sensitivity analysis can be dependent on the distribution of input variables [67]. In this work we have performed global sensitivity analysis on ENISI using 25 input factors. Each experiment has 250 time steps. Each simulation time step represents six hours in real world. For each factor, there are four different levels. Note that if one conducts a full factorial design for the 25 factors, it requires 425 = 1.126 × 1015 runs of experiments. Running one experiment in ENISI can take ninety minutes on average. Clearly, it is not possible to consider all possible combinations of parameter settings. A sparse design with economic run size is needed to conduct the sensitivity analysis.
As an effective statistical technique for sensitivity analysis, design of experiments (DOE) provides a way to carry out simulation experiments systematically that takes into account parameter interactions [68, 69]. By using experimental design, one can systematically bring statistical aspects into the analysis of results [70, 71]. It helps understand the relationships between input parameters and outputs in the model. DOE originates from real world experimentation in agriculture and engineering, and the techniques can be transferred to experiments with computer simulation such as agent-based models [72]. In the context of agent-based models, sensitivity analysis using DOE can uncover details about model behavior, help to identify the relative importance of inputs, and provide a common basis for discussing simulation results.
In traditional DOE a fractional factorial design, which considers a fraction of all possible level combinations, is often used for the analysis [73]. However, for a problem with a large number of parameters, the number of runs of the fractional factorial design is still large. To overcome the drawback of using a fractional factorial design, we utilize an experimental design strategy using an orthogonal array [65, 68, 73] to obtain a sparse design with desirable properties. The designs of orthogonal arrays often have small run sizes, but can accommodate all possible level combinations for at least any two columns. Specifically, an orthogonal array (OA) with strength t in experimental design is a design matrix such that for every t column, the possible distinct rows all appear the same number of times. That is, when projecting the designs in OA onto any subspace with dimensionality t, the coverage of parameter space is the same as that using full factorial design in t-dimensional space. Such a property maintains a good balance of level assignment for each factor. A random design will not guarantee this property [65, 68, 73, 74]. Thus, sensitivity analysis based on a design of orthogonal array enables the comprehensive investigation of parameters without dramatically increasing the number of simulation runs. Designs of orthogonal array have been widely used in many engineering applications [74] and sensitivity analysis of computer models [75]. For the sensitivity analysis of ENISI, we consider an orthogonal array of 128 runs on 25 factors with strength 2 (see S2 File for the complete design). The details of this design can be seen in [74]. This design has the following attractive properties: (i) projecting the design points onto any factor, there are exactly 16 replicates for each level; (ii) projecting the design points onto any two variables (i.e., any two columns of the design matrix), it is a full factorial (4 × 4) with 8 replications for each level combination. Fig 1 illustrates an OA design with 128 runs in 25-dimensional space. When it project points to a three-dimension subspace (labeled as x1, x2, and x3), the projected points consist of full factorial design in any two dimensional space of (x1, x2), (x1, x3), and (x2, x3). A random design will not be able to achieve this property. Hence, the proposed design is sparse in the sense that there are only 128 points in a 25-dimensional space, while the design points are well spread out with good properties. Therefore, the proposed design enables the effective study of the main effect of any factor.
projecting the design points onto any two-dimensional subspace, the projected points always form a full factorial (4 × 4) with 8 replications for each level of combination.
Experiments and Analysis.
We used a high performance computing cluster of 48 Intel Sandy Bridge nodes. Each node consists of two dual-socket Intel Sandy Bridge E5-2670 2.60GHz 8-core processors (16 cores per node) and 64GB of 1600MHz DDR3 RAM. The nodes are linked by QLogic QDR InfiniBand interconnects. For the MPI based implementation of our algorithms, we used the MPICH2 (version 1.7), which is optimized for QLogic InfiniBand cards. For each experiment we used 48 processors. On average it took ninety minutes to complete one simulation. Our cluster allowed us to run around five simulations in parallel.
According to the design of experiment we selected 128 sets of distinct parameter settings. As the model is stochastic in nature, there is inherent uncertainty in the output even though all the input parameters are kept same. Therefore we repeated the experiment for every parameter settings 15 times. In total 128 × 15 = 1920 simulation runs were conducted for global sensitivity analysis. A summary of the experiments is given in Table 3.
Results
Aim 1: Determine the influence of parameters by performing global sensitivity analysis using partial factorial experiments. An analysis of variance (ANOVA) is also performed to compute the level of significance for every parameter. Aim 2: Ranking the parameters (from greatest to least effect) from Aim 1, followed by studying the underlying causes with biological validation. Aim 3: Fine tuning of parameter values by applying local sensitivity analysis to the model.
Global sensitivity analysis determines the influence of a modeling parameter
The first attempt is to determine if there are any patterns of parameter influence on the system. To achieve this, we performed global sensitivity analysis of the system by following the design described in the “Materials and Methods” section. In this design each modeling parameter takes four monotonically increasing discrete values called “Levels.” Thus every parameter has four values: Level 1 to 4.
ENISI provides the total number of agents per phenotype in each time step as the system output. Typically these output are represented as time series plots, where the x and y axes represent time step and the number of cells respectively. To quantify the influence of any parameter on the model output, we generated time series plots for each input parameter. The time series plot for a parameter has four curves, each belonging to a distinct parameter Level. As shown in the design diagram, a total of 128 experiments with different parameter settings were performed, where each experiment was replicated 15 times. According to the experimental design, each parameter has 32 distinct experiments per level. Hence, each curve of the time series plot has 32 × 15 = 480 simulation runs. The average of the runs are drawn along with the associated error bars in each curve. For example, Fig 2(a) shows the time series curves (along with error-bars) of iTreg cells in the gastric lymph node (GLN) for modeling parameter vT.
Though these curves provide intuitive visual representation, comparisons between curves are difficult. These curves also fail to show the changes of system output when the levels of the corresponding modeling parameter are changed. Therefore, we used a different representation called a “main effect” plot (see Fig 2(b)). In a main effect plot, the x-axis represents the levels of the modeling parameter. The y-axis represents the average number of cells per level over the entire time period. Note that each experiment has 250 time steps and is replicated 15 times for each level. The average number of cells per level is determined from the average of all these experiments over the 250 time steps for the main effect plot. In this way, a time series curve is converted into a scalar value. Fig 2(b) shows the main effect plot of iTreg phenotype in the gastric lymph node (GLN) for parameter vT. From this, it is easy to see that with increasing levels of vT, the average number of iTreg phenotypes in GLN increases in a monotonic fashion. Though the pattern of this particular parameter is clear, there exist other patterns of main effect plots that are more challenging to interpret. We will address these patterns herein.
Main effect plots can be classified into four distinct patterns.
The analysis of the main effect plots for every output reveals four distinct patterns. They are: 1) monotonic, 2) bell shaped, 3) sigmoid, and 4) complex patterns. These are shown in Fig 3.
The monotonic pattern in Fig 3(a) shows that increasing the values of the parameter aT, results in the average cell count going up (or down) monotonically. The bell shaped pattern (Fig 3(b)) shows the average number of cells in the middle of the parameter range are significantly higher (or lower) than the extreme ends of the parameter range. The sigmoid pattern is shown in Fig 3(c). Notice that between Level 2 and Level 3 the average number of cells are changed dynamically but Level 2 & Level 1 and Level 3 & Level 4 have almost the same average number of cells. Thus, sigmoid patterns can be considered a switch between two consecutive parameter levels. The complex pattern in Fig 3(d) shows the average number of cells vary significantly between any two consecutive parameter levels, thereby forming a zig-zag shaped complex pattern.
Monotonic main effect patterns have significant influence over the system.
One of the prominent effects of a parameter on a complex system is monotonic in nature. In this case, with the increase of a parameter we observe monotonic increase or decrease of the model output. Fig 4 shows the time series and main effect plot of induced regulatory T cells in the gastric lamina propria for the parameter aT. Parameter aT represents the stimulation rate of naïve T cells into active cells. With the increase of aT the population of Treg cells is increased. From the main effect plot, we can clearly observe that with the increase of parameter aT = 0 (Level 1) to aT = 1 (Level 4), the population of Treg cells are also increased monotonically. Monotonic relationships of parameters are very common in immune responses and play crucial role in biological systems. Indeed, experimental data published in [64] shows an increase in numbers of the regulatory CD4+ T cells (Treg) subset during Helicobacter pylori infection. Based on these results, we could partially attribute the upregulation of the regulatory T cell population, among other effects, to the upregulation of the rate of T cell activation due to cytokine signaling [64].
Bell shaped patterns reveal interdependency among modeling parameters.
The effects of parameters can vary dramatically based on the parameter levels. Fig 5 shows the effect of parameter p17, which is the probability that a resting T cell differentiates to Th17 with the presence of inflammatory phenotypes. In the figure, on the boundary conditions (Level 1 and Level 4) the average population of Th17 is lower than the population simulations for Levels 2 and 3. This can be explained by the fact that, at p17 = 0 (Level 1), according to the model, resting T cells do not produce Th17, instead only inflammatory Th1 cells are produced. Th17 is only produced from the transition of iTreg to Th17. For this reason, we observe a lower number of Th17 cells for Level 1. On the other hand, at p17 = 1 (Level 4) all resting T cells produce only Th17 cells but not Th1. But, this differentiation requires inflammatory phenotypes such as Th1. Since no Th1 is directly produced from resting T cells by the model, there is a lack of inflammatory phenotypes to make the transition from resting T to Th17. For this reason, a lower number of Th17 cells is observed. For the intermediate levels (Level 2 & 3) a higher number of Th17 cells is observed as there is a balance between the rate of resting T cells to Th17 cell transition and an adequate number of Th1 cells as activators. This depicts one of the complex behaviors of parameter levels, which are typically bell shaped or inverted bell shaped in the model output.
Sigmoid main effect patterns exhibit switching behavior.
Minor differences in parameter levels play crucial roles in delineation of model output. For instance, in Fig 6, when the parameter ir’s value is changed from Level 1 to Level 2 we observe a dramatic change. However, we do not observe changes when Levels 3 or 4 are incremented. It appears that the effect of parameter value of ir is saturated after Level 2. Hence the significant value of ir must be within Levels 1 and 2. Parameter ir is the inhibitory factor of iTreg to Th17 transition. The Treg population is predominant over the Th17 cells during H. pylori infection. Hence, for lower values, a slight change of ir parameter has a significant effect on the system. Once the regulatory T cell population is significantly increased, the difference observed in these values is lower for values higher than 1, as the probability of transition from iTreg to Th17 is already very low. This kind of behavior results in a sigmoid main effect plot. Thus, a dramatic transition between two parameter values are observed; however, other values beyond the two levels do not have significant effect on the system output.
The behaviors of complex patterns are not well understood.
A fourth pattern is also observed in this study. With the increase of parameter values from Level 1 to Level 4, the number of cells increases and decreases creating a zig-zag shaped pattern in the main effect plot (see Fig 7). For example, consider the effect of parameter μCE on iTreg cells in lamina propria. Parameter μCE is the probability that pro-inflammatory epithelial cells are damaged by the inflammatory factors. By moving from Level 1 to Level 2, it is observed that the population of iTreg cells falls drastically and rises again for Level 3. The cell population falls again for Level 4. This type of parameter behavior requires further investigation.
ANOVA based analysis of global sensitivities
For the output immune responses and its 25 input parameters, it is important to investigate the underlying causes for how the parameters affect model outputs. To determine the impact of each parameter on the ENISI model, analysis of variance (ANOVA) was performed to find the most significant parameters. The key idea behind ANOVA use here is to link output and input through a statistical model with a normally distributed random error term. Based on the data, one can fit the model and quantify how the total variation of the output can be decomposed to variations contributed by estimated parameters, thus explaining the effects of sensitivity of input parameters.
Note that there are 128 different parameter setting for global sensitivity analysis, and each setting is replicated for 15 times in simulations. Therefore, there are 128 × 15 = 1920 data points for the ANOVA. For the output, the average count of each cell type over a time period (e.g., the entire time period or a time segmentation) as the response was used. In this study, by applying the ANOVA including the main effects, the observed variance in the output response is partitioned into components attributable to the 25 different parameters. Such an analysis provides a statistical quantification of whether or not each parameter plays a vital role to contribute the observed total variation of response output. Specifically, the significance of a parameter contributing to the total variation is examined by the statistical hypothesis testing using F statistic [76]. The F statistic is obtained by computing the ratio between the variation of output explained by the parameter and the total variation of response output. Thus, a large value of F statistic indicates a strong evidence of the significance of the parameter. The p-value based on the F test can be calculated accordingly [77], having small values with large F statistic. A statistically significant result with p-value lower than a user-defined threshold (i.e. significance level) justifies the rejection of the null hypothesis that the parameter is insignificant. Here the significant level is set to be 0.05. Recall that the design used for the experiment is an orthogonal array with 128 runs. Such a design makes each column of the design matrix mutually orthogonal to each other. As a result, the ANOVA based on such a design allows the estimated main effects to not correlate with each other. It provides an efficient technique to determine the sensitivity of each parameter with respect to the output of cell counts.
To ensure the validity of the ANOVA results, we also examine the model assumptions of the ANOVA approach. Note that the ANOVA method assumes that the independent and identically distributed error term exhibits normal distribution. For the global sensitivity analysis, we use the residual scatter plot to examine the independence of the data, and examine the normality assumption of the error terms through the histogram of residuals and the QQ-normal plot. The scatter plot, histogram, and QQ-normal plots of the residuals for the nine cells types as responses are shown in S1 Fig. As we can see from the figures, the independence and normality assumptions are generally valid when using the ANOVA for the global sensitivity analysis for our data. It is worth pointing out there are some potential outliers existing in the data, which could distort the model assumptions to some extent. Detailed investigation of those abnormal points with domain experts are often recommended before removing these points from further analysis.
Overall significance of modeling parameters.
Using the technique described above, ANOVA analysis was performed to calculate the p-values of significance of a parameter for each of the nine output variables. This is shown in Fig 8. The x-axis represents the 25 parameters and the y-axis represents the nine output cell types.
From Fig 8 it is clear that some parameters, such as aT, vT, vEC, are significantly important for each cell type. There are certain parameters that are significant only for T cell populations (e.g. p17, y17) and some that are responsible for macrophages (e.g. br).
The global sensitivity analysis shows which parameters are most important for model tuning. Hence, it is also helpful to identify the parameters that have the smallest effects on the model. For example it is clear that parameter a17 and i17 have little effect on the system.
Temporal significance of modeling parameters.
In addition to measuring sensitivity analysis for a parameter over the entire time period, it is possible to analyze in which time period a parameter is most significant. This is particularly important because it allows investigation of the significance of a parameter over time.
For this analysis we divided the entire time period into nine segments, each representing one experimental week. The one experimental week time interval is chosen from a biological point of view, as we are interested to see how the inflammation progresses per week. Also, in the in vivo experiments, biological data is collected weekly, which allows us to compare and validate simulated and biological results. Next, the significance of a parameter within the nine week period was plotted. The significance of a parameter is represented by the p-value for the changes of the parameter determined using ANOVA. The results for parameters vEC, aT and p17 are given in Fig 9. Major changes in the dynamic significance of parameters were identified. In part the ANOVA findings are intuitive and serve as a measure to ensure that the model behaves in a biologically relevant manner. For instance, the probability of resting T cell stimulation, aT, has a high impact on all T cell subsets and activated macrophages with increasing significance over time. This is explained primarily by T cell population density during the course of infection. At the beginning of infection, Th1 and Th17 populations are low in contrast to when infection progresses, which is when these cell types become more critical for proliferation and differentiation of additional T cells in the gastric lamina propria (Fig 9).
One interesting finding is seen in the behavior of epithelial cells in response to inflammatory factors. Specifically, the probability that an epithelial cell transitions into a proinflammatory state as determined by parameter vEC has a higher impact on overall immunity when compared to epithelial damage (br). In fact, vEC is the single most significant parameter in the model as determined by ANOVA. These results suggest that epithelial cells that come in contact with H. pylori significantly influence the overall immune response. Moreover, the temporal effects of vEC increase as infection progresses and don’t approach the significant p-value bracket of 0.01–0.1 until at least 4 weeks post-infection, implying that this response wouldn’t be detected until late after infection (Fig 9c). Although out of the scope of this paper, a longer time course simulation should be performed to predict at which time frame this epithelial cell property is most crucial to host immunity to H. pylori.
Selection of most significant parameters.
In order to refine parameter selection we need to determine the most significant parameters based on their impact on model dynamics. To rank the parameters we use two different systems using global sensitivity analysis. First we take average p-values for each of the output cell types and order them from smallest to largest. In this way, we find that the top ten significant parameters are: νEC, aT, νT, a2 y2, a1, i2, νBs, p17, and br. We also ranked the parameters based on how many outputs are significant for each parameter. For this type of ranking we find that: aT, νT, νEC, y2, p17, i2, y17, a2, a1, and νBs are the top ten significant parameters. Both lists are almost identical other than br and y17. Parameter y17 can be considered more important if p-values and the number of significant output parameters are compared.
Local sensitivity analysis
Global sensitivity analysis measures the significance of each parameter. We can rank the most significant parameters based on p-values as described in the previous section. We choose the top ten significant parameters to analyze their local sensitivity behavior. The ten most significant parameters along with the corresponding test values are shown in Table 4.
To study the effect of a single parameter over the population, we created a base set of parameter values called the base configuration from studies in literature and experimental calibration process. As the global sensitivity results show the region of interest, we selected three new different values from the region. Thus every parameter has four different values. Table 4 lists all the parameters and the tested values.
In the following sections, the local effects of the top two significant parameters p17 and vEC along with a biologically interesting parameter μM1 are described in detail.
Parameter p17 control the T cell dynamics significantly.
Parameter p17 is the probability of resting T cell stimulation to Th17. The default value for the model is set at 0.1. We also observed the output for values of 0.25, 0.5, and 0.9. The experimental results are shown in Fig 10. When the value of p17 is increased, resting T cells are more likely to produce Th17Prolif than Th1Prolif. So, production of Th17 is increased. We can see that with increasing p17 we should create lower numbers of Th1 as compared to Th17. We also implemented transition of resting T to contact dependent transition. The transition depends on eDC and M1. As p17 does not directly affect M1 or eDC, we do not observe much change in the dynamics of M1 and eDC for p17. So, the effect of parameter p17 for the Th1, Th17 and iTreg dynamics are not very sensitive.
Parameter vEC has very high sensitivity.
Parameter vEC is the probability that EC cells transition into pEcell. The experimental results are shown in Fig 11. The default value of vEC is 0.01. This is the lowest probability value of the parameter that was tested. When we increased the probability to a slightly higher value of 0.1, we observed significant change. Increasing vEC from 0.01 to 0.1 increases iTreg cells at the GLN, and M0, M1, and M2 macrophages in the LP primarily, and exerts a lesser impact in Th1 and Th17 cell subsets. A recent paper implicates Sonic Hedgehog, a regulator of gastric epithelial cell differentiation and function, which is increased after H. pylori infection [78], as critical for macrophage recruitment to the stomach LP early after H. pylori infection [79].
Parameter μM1 controls macrophage dynamics.
Parameter μM1 represents the ability for pro-inflammatory M1 cells to eliminate bacteria. The default value of μM1 is 1. To observe the effect of μM1 on the system, the value was changed to 0.5, 0.25 and 0.0 in decreasing order such that fewer bacteria are killed. The ability for macrophages to rid the system of H. pylori caused the most noticeable changes in a peak of lamina propria myeloid cells (M0) between days 10 and 22 (Fig 12). More than a 10-day delay occurs in the M0 population when μM1 from is shifted from 0.25 to 0.5. Additionally, at the higher killing capacity (μM1 = 0.5) bacteria is cleared more rapidly but causes more significant inflammation when compared to μM1 = 0.25. When μM1 is set to its default value, the myeloid population peaks between days 15 and 16 and prevents an overzealous cellular response.
Interpretation of sensitivity analysis results as new hypothesis
Based on the sensitivity analysis, νEC is the most critical factor in the system’s dynamics. This finding indicates that perturbations induced by contact of H. pylori with epithelial cells are the major drivers of the events and reactions that characterize the immune response to the bacteria. The great majority of H. pylori in the gastric mucosa is found free-floating in the mucus layer and associated with the apical side of epithelial cells [80, 81]. Thus, the model recapitulates this principle and the sensitivity analysis indicates that the epithelium is the compartment which receives the major impact of the infection. An interesting and less intuitive finding of the model is the relationship between epithelium and macrophages. Based on analysis across νEC ranges, activation of epithelial cells by H. pylori leads to a sequential accumulation of macrophage precursors (M0) between days 0 and 14/21 post-infection. This first wave is followed by a subset of M2 with regulatory/suppressor role, which peaks between days 21 and 35 post-infection. Starting at day 45 there is a predominance of M1 pro-inflammatory macrophages. This sequence of changes is caused by increased probability of epithelial cells transitioning to a proinflammatory state, which should directly induce a proinflammatory macrophage. It will be interesting to screen for the presence of regulatory macrophages in the gastric mucosa following H. pylori infection and evaluate their role in the outcome of the infection.
Discussion
Sensitivity analysis of a large scale agent-based model is challenging due to complex model structure and expensive computation needed to run a full factorial experiment in silico. The running time of the models prohibits the consideration of a full factorial combination of parameters for global sensitivity analysis. In this work, we have developed a novel and comprehensive sensitivity analysis by using an effective experimental design strategy. In spite of a large number of parameters and a continuous space of parameter values, the proposed method can capture the global nature of the system and gain proper knowledge of the significance of parameters on the system.
The proposed experimental methods tackle the challenges faced in obtaining meaningful results and proper insight into the model. The design of experiments only needs an economic run size in order to obtain the proper sensitivity analysis on a global scale. The results also showed that the design, with four levels for each parameter, captures some interesting main-effect shapes. We observed the various shapes of main-effect plots and classified them into four general categories. Meaningful explanations are provided to illustrate the aptness of these shapes in the ENISI model with validation from biological reality.
The global sensitivity analysis guided us as to the range of parameter values. The model was then calibrated to a base parameter setting using biological literature and findings from in vivo experiments [64]. To further fine tune the model, we performed local sensitivity analysis of the ten most important parameters. We have varied the values of the parameters around the base value to observe changes in output. Whenever the outputs were deemed closer to experimental results, the parameter value was updated. With a few iterations, we achieved a calibrated model of the mucosal immune responses to H. pylori infection.
In this study, the global sensitivity analysis of ENISI revealed four distinct patterns for the main effects of the input parameters. For the monotonic pattern for a parameter, its contribution to the output is easy to understand and can be controlled by simply increasing or decreasing the parameter value. Bell shaped patterns reveal complex inter-dependency among more than one parameter. Also the complex zig-zag shaped patterns for a parameter are not yet well understood and require further investigation. Future work can investigate more on inter-dependencies and relationships among the parameters. Orthogonal arrays provided a natural way to capture pairwise interactions. The interaction effect plots can show the interaction of parameters with the system and their interdependencies. If the focus of the study is on three-way or higher order interactions, one can consider orthogonal arrays with a strength larger than two or some continuous-based design such as Latin hypercube [82, 83].
In this work, the experiment and analysis are performed on the H. pylori model implemented using the ENISI platform, an agent-based model. However, the proposed methodology is general in nature, and can work for other complex simulation systems with a large number modeling parameters, such as models in computational epidemics, social network analysis, etc. The proposed method can also be a useful tool for validation and parameter estimation with a limited budget in terms of experiment time and effort. The temporal sensitivity analysis also reveals the time periods when the system is sensitive to other parameters. To calibrate the computational model with some existing systems, it can be useful to collect experimental data in the specific time period of interest.
Supporting Information
S1 Fig. Residual scatter, histogram, and QQ-normal plots for nine output parameters.
https://doi.org/10.1371/journal.pone.0136139.s002
(TIF)
S1 File. Values of parameters for each of the four levels.
https://doi.org/10.1371/journal.pone.0136139.s003
(XLSX)
S2 File. Orthogonal array of 128 experiment runs on 25 factors.
https://doi.org/10.1371/journal.pone.0136139.s004
(XLSX)
Acknowledgments
We thank our external collaborators and members of the Network Dynamics and Simulation Science Laboratory (NDSSL) and the Nutritional Immunology and Molecular Medicine Laboratory (NIMML, http://www.nimml.org) for their suggestions and comments. This work has been partially supported by Defense Threat Reduction Agency (DTRA)—R&D (HPC) Award No. HDTRA1-09-1-0017, DTRA—Validation Award No. HDTRA1-11-1-0016, DTRA—Comprehensive National Incident Management System (CNIMS) Award No. HDRTA1-11-D-0016-0001, National Science Foundation (NSF) PetaApps Grant OCI-0904844, NSF Network Science and Engineering (NetSE) Grant CNS-1011769, NSF Software Development for Cyberinfrastructure (SDCI) Grant OCI-1032677, National Institutes of Health (NIH) MIDAS project 2U01GM070694-7 and National Institute of Allergy and Infectious Diseases (NIAID) & NIH project HHSN272201000056C.
Author Contributions
Conceived and designed the experiments: MA XD MM. Performed the experiments: MA XD. Analyzed the data: MA XD CP AC RH JBR MM. Contributed reagents/materials/analysis tools: SE KB SH YM VA. Wrote the paper: MA XD VA RH MM.
References
- 1.
Saltelli A, Annoni P, Azzini I, Campolongo F, Ratto M, Tarantola S. Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Computer Physics Communications. 2010;.
- 2.
Helton JC. Uncertainty and Sensitivity Analysis for Models of Complex Systems. In: Computational Methods in Transport: Verification and Validation. Springer Berlin Heidelberg; 2008.
- 3.
Cacuci DG, Ionescu-Bujor M, Navon IM. Sensitivity And Uncertainty Analysis: Applications to Large-Scale Systems. Chapman & Hall/CRC; 2005.
- 4. Brooks RJ, Semenov MA, Jamieson PD. Simplifying Sirius: sensitivity analysis and development of a meta-model for wheat yield prediction. European Journal of Agronomy. 2001;.
- 5.
Do H, Rothermel G. Using Sensitivity Analysis to Create Simplified Economic Models for Regression Testing. In: Proceedings of the 2008 International Symposium on Software Testing and Analysis; 2008.
- 6. Sher AA, Wang K, Wathen A, Maybank PJ, Mirams GR, Abramson D, et al. A local sensitivity analysis method for developing biological models with identifiable parameters: Application to cardiac ionic channel modelling. Future Generation Computer Systems. 2013;.
- 7. Frey HC, Patil SR. Identification and Review of Sensitivity Analysis Methods. Risk Analysis. 2002;.
- 8. Hamby DM. A comparison of sensitivity analysis techniques. Health Physics. 1995;. pmid:7814253
- 9.
Saltelli A, Tarantola S, Campolongo F, Ratto M. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. Halsted Press; 2004.
- 10. Arwade SR, Moradi M, Louhghalam A. Variance decomposition and global sensitivity for structural systems. Engineering Structures. 2010;.
- 11. Bauer KW, Parnell GS, Meyers DA. Response surface methodology as a sensitivity analysis tool in decision analysis. Journal of Multi-Criteria Decision Analysis. 1999;.
- 12. Lim JT, Gold HJ, Wilkerson GG, Raper JCD. A Monte Carlo/response surface strategy for sensitivity analysis: application to a dynamic model of vegetative plant growth. Appl Math Model. 1989;. pmid:11538114
- 13. Helton JC, Johnson JD, Sallaberry CJ, Storlie CB. Survey of sampling-based methods for uncertainty and sensitivity analysis. Reliability Engineering & System Safety. 2006;.
- 14.
Bonabeau E, Dorigo M, Theraulaz G. Swarm Intelligence: From Natural to Artificial Systems. Oxford University Press, Inc.; 1999.
- 15. Dong X, Foteinou PT, Calvano SE, Lowry SF, Androulakis IP. Agent-Based Modeling of Endotoxin-Induced Acute Inflammatory Response in Human Blood Leukocytes. PLoS ONE. 2010;.
- 16. Forrest S, Beauchemin C. Computer immunology. Immunol Rev. 2007;. pmid:17367343
- 17. Neugebauer EA, Willy C, Sauerland S. Complexity and non-linearity in shock research: reductionism or synthesis? Shock. 2001;.
- 18. An G. Concepts for developing a collaborative in silico model of the acute inflammatory response using agent-based modeling. J Crit Care. 2006;. pmid:16616634
- 19. Lowry SF, Calvano SE. Challenges for modeling and interpreting the complex biology of severe injury and inflammation. J Leukoc Biol. 2008;. pmid:17984288
- 20. Seely AJ, Christou NV. Multiple organ dysfunction syndrome: exploring the paradigm of complex nonlinear systems. Crit Care Med. 2000;. pmid:10921540
- 21. An G, Faeder J, Vodovotz Y. Translational systems biology: introduction of an engineering approach to the pathophysiology of the burn patient. J Burn Care Res. 2008;.
- 22. Vodovotz Y, Chow CC, Bartels J, Lagoa C, Prince JM, Levy RM, et al. In silico models of acute inflammation in animals. Shock. 2006;.
- 23. Baldazzi V, Castiglione F, Bernaschi M. An enhanced agent based model of the immune system response. Cellular Immunology. 2006;. pmid:17416357
- 24. Mi Q, Rivière B, Clermont G, Steed DL, Vodovotz Y. Agent-based model of inflammation and wound healing: insights into diabetic foot ulcer pathology and the role of transforming growth factor-β1. Wound Repair and Regeneration. 2007;. pmid:17971013
- 25. An G. Agent-based computer simulation and sirs: building a bridge between basic science and clinical trials. Shock. 2001;. pmid:11580108
- 26. An G. Introduction of an agent-based multi-scale modular architecture for dynamic knowledge representation of acute inflammation. Theoretical Biology & Medical Modelling. 2008;.
- 27. Shapiro M, Duca KA, Lee K, Delgado-Eckert E, Hawkins J, Jarrah AS, et al. A virtual look at Epstein-Barr virus infection: simulation mechanism. J Theor Biol. 2008;. pmid:18371986
- 28. Duca KA, Shapiro M, Delgado-Eckert E, Hadinoto V, Jarrah AS, Laubenbacher R, et al. A virtual look at Epstein-Barr virus infection: biological interpretations. PLoS Pathog. 2007;. pmid:17953479
- 29. Bauer AL, Beauchemin CAA, Perelson AS. Agent-based modeling of host-pathogen systems: The successes and challenges. Information Sciences. 2009;. pmid:20161146
- 30.
Barrett, CL, Bisset, KR, Eubank, SG, Feng, X, Marathe, MV. EpiSimdemics: an Efficient Algorithm for Simulating the Spread of Infectious Disease over Large Realistic Social Networks. In: SC’08: ACM/IEEE Conference on Supercomputing; 2008.
- 31.
Barrett CL, Bisset K, Chen J, Eubank S, Lewis B, Kumar VSA, et al. Interactions among human behavior, social networks, and societal infrastructures: A Case Study in Computational Epidemiology. In: Fundamental Problems in Computing. Springer Netherlands; 2009.
- 32. Eubank S, Guclu H, Kumar VSA, Marathe M, Srinivasan A, Toroczkai Z, et al. Modelling disease outbreaks in realistic urban social networks. Nature. 2004;. pmid:15141212
- 33. Barrett C, Beckman R, Berkbigler K, Bisset K, Bush B, Campbell K, et al. TRANSIMS: Transportation Analysis Simulation System. LANL; 2001.
- 34. Barrett E, Ross RP, Fitzgerald GF, Stanton C. Rapid screening method for analyzing the conjugated linoleic acid production capabilities of bacterial cultures. Appl Environ Microbiol. 2007;.
- 35. Karaoz U, Murali TM, Letovsky S, Zheng Y, Ding C, Cantor CR, et al. Whole-genome annotation by using evidence integration in functional-linkage networks. Proceedings of the National Academy of Sciences of the United States of America. 2004;. pmid:14981259
- 36. Wendelsdorf KV, Bassaganya-Riera J, Bisset K, Eubank S, Hontecillas R, Marathe M. ENteric Immunity Simulator Version 0.9: A tool for in silico study of gut immunopathologies. NDSSL; 2011.
- 37.
Wendelsdorf KV, Bassaganya-Riera J, Bisset K, Eubank S, Hontecillas R, Marathe M. ENteric Immunity Simulator: A tool for in silico study of gut immunopathologies. Proceedings of the IEEE International Conference Bioinformatics and Biomedicine. 2011;.
- 38.
Bisset K, Alam M, Bassaganya-Riera J, Carbo A, Eubank S, Hontecillas R, et al. High-Performance Interaction-Based Simulation of Gut Immunopathologies with ENteric Immunity SImulator (ENISI). In: Parallel Distributed Processing Symposium (IPDPS), 2012 IEEE 26th International; 2012.
- 39. Wendelsdorf KV, Alam M, Bassaganya-Riera J, Bisset K, Eubank S, Hontecillas R, et al. ENteric Immunity SImulator: a tool for in silico study of gastroenteric infections. IEEE Trans Nanobioscience. 2012;. pmid:22987134
- 40.
Mei Y, Hontecillas R, Zhang X, Bisset K, Eubank S, Hoops S, et al. ENISI Visual, an agent-based simulator for modeling gut immunity. In: Bioinformatics and Biomedicine (BIBM), 2012 IEEE International Conference on; 2012.
- 41.
Mei Y, Carbo A, Hontecillas R, Hoops S, Liles N, Lu P, et al. ENISI MSM: A novel multi-scale modeling platform for computational immunology. In: Bioinformatics and Biomedicine (BIBM), 2014 IEEE International Conference on; 2014.
- 42. Mei Y, Abedi V, Carbo A, Zhang X, Lu P, Philipson C, et al. Multiscale modeling of mucosal immune responses. BMC Bioinformatics. 2015;.
- 43. Kelly D, Campbell JI, King TP, Grant G, Jansson EA, Coutts AG, et al. Commensal anaerobic gut bacteria attenuate inflammation by regulating nuclear-cytoplasmic shuttling of PPAR-gamma and RelA. Nat Immunol. 2004;. pmid:14691478
- 44. Marzo AL, Klonowski KD, Bon AL, Borrow P, Tough DF, Lefrancois L. Initial T cell frequency dictates memory CD8+ T cell lineage commitment. Nat Immunol. 2005;. pmid:16025119
- 45. Lanzavecchia A, Sallusto F. Regulation of T cell immunity by dendritic cells. Cell. 2001;. pmid:11509174
- 46. Sallusto F, Geginat J, Lanzavecchia A. Central memory and effector memory T cell subsets: function, generation, and maintenance. Annu Rev Immunol. 2004;. pmid:15032595
- 47. Kaech SM, Ahmed R. Immunology: CD8 T cells remember with a little help. Science. 2003;. pmid:12690179
- 48. Naresh R, Song Y, Hampson DJ. The intestinal spirochete Brachyspira pilosicoli attaches to cultured Caco-2 cells and induces pathological changes. PLoS One. 2009;. pmid:20020053
- 49. Carbo A, Hontecillas R, Kronsteiner B, Viladomiu M, Pedragosa M, Lu P, et al. Systems modeling of molecular mechanisms controlling cytokine-driven CD4+ T cell differentiation and phenotype plasticity. PLoS Comput Biol. 2013;. pmid:23592971
- 50. Carbo A, Hontecillas R, Andrew T, Eden K, Mei Y, Hoops S, et al. Computational modeling of heterogeneity and function of CD4+ T cells. Front Cell Dev Biol. 2014;. pmid:25364738
- 51. Carbo A, Olivares-Villagomez D, Hontecillas R, Bassaganya-Riera J, Chaturvedi R, Piazuelo MB, et al. Systems modeling of the role of interleukin-21 in the maintenance of effector CD4+ T cell responses during chronic Helicobacter pylori infection. MBio. 2014;. pmid:25053783
- 52. Carbo A, Hontecillas R, Viladomiu M, Leber A, Hoops S, Bassaganya-Riera J. Modeling the Dynamics of T Helper 17 Induction and Differentiation. MOJ Immunol. 2015;.
- 53. Leber A, Viladomiu M, Hontecillas R, Abedi V, Philipson C, Hoops S, et al. Systems modeling of interactions between mucosal immunity and the gut microbiome during Clostridium difficile infection. PLoS One. 2015;. pmid:26230099
- 54. Sumner T, Hetherington J, Seymour RM, Li L, Rey MV, Yamaji S, et al. A composite computational model of liver glucose homeostasis. II. Exploring system behaviour. J R Soc Interface. 2012;.
- 55. Sumner T, Shephard E, Bogle ID. A methodology for global-sensitivity analysis of time-dependent outputs in systems biology modelling. J R Soc Interface. 2012;.
- 56.
Sumner T. Sensitivity analysis in systems biology modelling and its application to a multi-scale model of blood glucose homeostasis. University College London; 2010.
- 57. Broeke GAT, Voorn GAKv, Ligtenberg A. Sensitivity analysis for agent-based models: a low complexity test-case. Laboratory of Geo-information Science and Remote Sensing; 2014.
- 58. Schouten M, Verwaart T, Heijman W. Comparing two sensitivity analysis approaches for two scenarios with a spatially explicit rural agent-based model. Environmental Modelling & Software. 2014;.
- 59. Ligmann-Zielinska A, Kramer DB, Cheruvelil KS, Soranno PA. Using Uncertainty and Sensitivity Analyses in Socioecological Agent-Based Models to Improve Their Analytical Performance and Policy Relevance. PLoS ONE. 2014;. pmid:25340764
- 60. Parry HR, Topping CJ, Kennedy MC, Boatman ND, Murray AWA. A Bayesian sensitivity analysis applied to an Agent-based model of bird population response to landscape change. Environmental Modelling & Software. 2013;.
- 61. Fonoberova M, Fonoberov VA, Mezić I. Global sensitivity/uncertainty analysis for agent-based models. Reliability Engineering & System Safety. 2013;.
- 62. Kandulski A, Malfertheiner P, Wex T. Role of regulatory T-cells in H. pylori-induced gastritis and gastric cancer. Anticancer Res. 2010;. pmid:20530414
- 63. Ramarao N, Meyer TF. Helicobacter pylori resists phagocytosis by macrophages: Quantitative assessment by confocal microscopy and fluorescence-activated cell sorting. Infection and Immunity. 2001;. pmid:11254625
- 64. Carbo A, Bassaganya-Riera J, Pedragosa M, Viladomiu M, Marathe M, Eubank S, et al. Predictive Computational Modeling of the Mucosal Immune Responses during Helicobacter pylori Infection. PLoS ONE. 2013;. pmid:24039925
- 65.
Wu CJ, Hamada MS. Experiments: planning, analysis, and optimization. John Wiley & Sons; 2011.
- 66.
Happe, K. Agent-based modelling and sensitivity analysis by experimental design and metamodelling: an application to modelling regional structural change. In: XIth International Congress of the European Association of Agricultural Economists; 2005.
- 67. Rodriguez-Fernandez M, Banga JR, Doyle FJ. Novel global sensitivity analysis methodology accounting for the crucial role of the distribution of input parameters: application to systems biology models. International Journal of Robust and Nonlinear Control. 2012;.
- 68.
Box GE, Hunter JS, Hunter WG. Statistics for experimenters. Wiley New York; 2005.
- 69.
Kleijnen JPC, Groenendaal Wv. Simulation: A Statistical Perspective. John Wiley & Sons, Inc.; 1992.
- 70.
Law AM, Kelton DM. Simulation Modeling and Analysis. McGraw-Hill Higher Education; 1999.
- 71. Noordegraaf AV, Nielen M, Kleijnen JPC. Sensitivity analysis by experimental design and metamodelling: Case study on simulation in national animal disease control. European Journal of Operational Research. 2003;.
- 72.
Kleijnen JP, Sanchez SM, Lucas TW, Cioppa TM. A user’s guide to the brave new world of designing simulation experiments; 2003.
- 73.
Montgomery DC. Design and analysis of experiments. John Wiley & Sons; 2012.
- 74.
Hedayat AS, Sloane NJA, Stufken J. Orthogonal arrays: theory and applications. Springer; 1999.
- 75. Morris MD, Moore LM, McKay MD. Using orthogonal arrays in the sensitivity analysis of computer models. Technometrics. 2008;.
- 76.
Scheffé H. The Analysis of Variance. Wiley; 1959.
- 77.
Seber GAF, Lee AJ. Linear Regression Analysis. Wiley; 2012.
- 78. Bassaganya-Riera J, Dominguez-Bello MG, Kronsteiner B, Carbo A, Lu P, Viladomiu M, et al. Helicobacter pylori Colonization Ameliorates Glucose Homeostasis in Mice through a PPAR γ-Dependent Mechanism. PLoS ONE. 2012;.
- 79. Schumacher MA, Donnelly JM, Engevik AC, Xiao C, Yang L, Kenny S, et al. Gastric Sonic Hedgehog Acts as a Macrophage Chemoattractant During the Immune Response to Helicobacter pylori. Gastroenterology. 2012;.
- 80. Kronsteiner B, Bassaganya-Riera J, Philipson N, Hontecillas R. Novel insights on the role of CD8+ T cells and cytotoxic responses during Helicobacter pylori infection. Gut Microbes. 2014;. pmid:24755940
- 81. Sigal M, Rothenberg ME, Logan CY, Lee JY, Honaker RW, Cooper RL, et al. Helicobacter pylori Activates and Expands Lgr5 Stem Cells Through Direct Colonization of the Gastric Glands. Gastroenterology. 2015;. pmid:25725293
- 82. Sacks J, Welch WJ, Mitchell TJ, Wynn HP. Design and analysis of computer experiments. Statistical science. 1989;.
- 83.
Santner TJ, Williams BJ, Notz W. The Design and analysis of computer experiments. Springer; 2003.