Provenance data model.

We consider the types and relations defined by the PROV data model: PROV-DM [19]. PROV-DM includes the following types: entity, activity, agent as well as the following relations: WasGeneratedBy, Used, WasInformedBy, WasDerivedFrom, WasAttributedTo, WasAssociatedWith, ActedOnBehalf-Of. Provenance information is usually depicted as a directed, acyclic graph where the arrowheads show towards the sources or predecessors of an entity, activity, or agent—thus, depicting its origin. For our case study, we are only focusing on the types entity and activity as well as on the relations WasGeneratedBy and Used. The reasons for our decision will be explained in the Results and discussion section.

Provenance ontology for simulation studies.

Recently, Ruscheinski et al. (2018) [20] have applied PROV-DM for capturing provenance information from entire simulation studies and initiated a definition of a PROV-DM ontology for these studies. Important ingredients of this ontology have been identified to be “a) specific types of entities (e.g., data, theories, simulation experiments, and simulation models), b) specific roles between these entities (e.g., used as input, used for calibration, used for validation, used for adaptation, used for extension, used for composition), c) specific refinement of activities (i.e., successive refinement of activities down to a level where simulation experiment specifications define activities and thus are ready to be executed), and d) specific inference strategies (e.g., warnings if the same data has been used both by calibration and validation activities, or the option to reuse validation experiments among model descendants to check consistency)” [20]. The adaptation and application of this ontology for capturing the essential information of our case study is presented in the Results and discussion section.

Collecting provenance information.

In order to gather all relevant information, the publications as well as the supporting materials—as they often contain model and experiment descriptions—were read thoroughly. Referenced publications were checked, as well, whenever they appeared to be important for the development of the simulation model. All information that resembled provenance entities were marked. While reading a study, a first sketch of a possible provenance graph was made. Afterwards, a revision of all markings helped to finalize the graph and to remove duplicate entities. Often, authors described their simulation study chronologically, which made it easy to determine the path of its development, but sometimes, the connections of the entities had to be inferred from the context. In general, tracing provenance information from an entire simulation study in retrospective involved some interpretation of the results presented in the publication.

Provenance nodes.

The main concept of WebProv is the Neo4j Provenance Node and the dependency graph created from related Provenance Nodes using Neo4j relationships. A Provenance Node represents an entity or activity and, therefore, must have a classification (e.g., Simulation Model or Building Simulation Model) which defines the types of relationships that can be formed with other nodes depending on our PROV ontology. For example, the Simulation Model entity can only be created by a Building Simulation Model or a Calibrating Simulation Model activity (see the Results and discussion section for details). These classifications can be easily changed or extended if necessary. Additionally, Study nodes store information about a particular study (a reference to a study and the name of the signaling pathway it is concerned with) and group a set of Provenance Nodes together. Finally, InformationField nodes allow us to attach zero or more key–value pairs to a Provenance Node to store further information. In our case, we describe which information should be entered depending on the entity type in the Results and discussion section.

Queries.

WebProv allows two types of queries: text queries and queries in Cypher—Neo4j’s query language. The text query field will perform a fuzzy search of the data contained within the nodes. If successful, a set of nodes are returned that contain the given phrase and the user can choose to add any of these nodes to the graph. Alternatively, the Cypher field passes the query as a string to the backend which forwards it to the Neo4j database. Since arbitrary queries can be performed, when the results are returned, the frontend attempts to parse the results using io-ts as a Provenance Node. If successful, these nodes are automatically shown on the graph. This method allows more complex queries, in particular, subgraphs to be extracted. Thus, structural information can be accessed.

Provenance entities.

For every Provenance Node (entity or activity), we require the following details to be provided: a) (PROV-DM) Type, b) Study (Reference), c) Description.

The (PROV-DM) Type declares the type of entity or activity. The Study (Reference) contains the name of the study a Provenance Node belongs to. The Description contains some textual explanation of the Provenance Node. For some entities, we are asking for further information, as seen in Fig 2, which we will elaborate on.

Research Question [RQ]:

The research question (or research objective or problem formulation) determines the goal of the research presented in a publication. For simulation studies, it typically forms the starting point of the modeling and simulation life cycle [1, 67] and is key to interpreting its outputs such as simulation data or a simulation model.

As for the provenance example shown in Fig 3, Lee et al. (2003) questioned in RQ1 the necessity of “the two scaffold proteins, APC and Axin” (for Wnt signaling) and whether “their roles differ” [28]. This research question determines a minimum number of model constituents and guides the modeler towards possible simulation experiments to be executed.

Assumption [A]:

We define assumptions of a simulation model to be all statements that refer to abstractions or specializations of the described model. Assumptions typically deal with the input of a model (e.g., assume the concentration of x to be constant or let the initial value of y be …) and may set the boundaries of the system under consideration or partially explain the thoughts behind a simulation model—always with the research question in mind.

In order to facilitate the analysis of assumptions, we adopted the Systems Biology Ontology (SBO) [68] to categorize the assumptions. SBO provides “structured controlled vocabularies, comprised of commonly used modeling terms and concepts” [69] and is primarily used to “describe the entities used in computational modeling (in the domain of systems biology)” [68]. By using SBO, we are trying to answer which part of the model contains assumptions rather than what was assumed.

In the provenance example, three assumptions with three different categories could be identified. A1, for instance, reads “Dsh, TCF, and GSK3β are degraded very slowly, we assume that their concentrations remain constant throughout the timecourse of a Wnt signaling event” [28] and was matched to ID 362 (Concentration conservation law) of SBO.

Requirement [R]:

Requirements define properties that the results of a simulation model need to show. These may be used for the purpose of calibrating or validating a simulation model. They also direct the modeler towards adaptation of a model if the requirements are not met. We do not consider other kinds of requirements (e.g., the need of using specific tools or approaches in performing a simulation study).

Typically, simulation data needs to be compared with real-world data—in our case wet-lab data. These real-world measurements determine the species of interest which should be part of the Requirement entity. Therefore, we record the main species considered by the requirement as well as its type (either qualitative or quantitative) and connect the Requirement to the wet-lab data it relates to. The list of main species will make it easier to compare, interrelate and reuse simulation models as they determine the focus of the model.

We were able to identify one requirement R1 in the provenance example of Lee et al. (2003). The quantitative requirement that “Axin stimulates the phosphorylation of β-catenin by GSK3β at least 24,000-fold” [28] actually refers to the wet-lab data WD1 obtained in another study by Dajani et al. (2003) [65]. Its main species are Axin, β-catenin, and GSK3β.

Qualitative Model [QM]:

We define the qualitative model to be a network diagram, such as a reaction scheme (chemical reaction network diagram), which contains the entities of the system (e.g., species) and their interactions. This diagram may be presented in a formal (e.g., using the Systems Biology Graphical Notation (SBGN) [70] or a Boolean network diagram [71]) or informal way. All textual descriptions of a simulation model that do not include quantitative information (e.g., reaction rate constants, initial values) can also be part of the qualitative model. It should be noted that the qualitative model is also called conceptual model [72] sometimes, whereas in other publications, the qualitative model forms part of the conceptual model [3].

We record a reference to the qualitative model, which, for example, could be a reference to a figure in the publication, or, ideally, a DOI. Furthermore, we denote a list of species and compartments used in the model. Multiple compartments require a shuttling of species from one compartment to another one and every compartment should, ideally, have an area (for the transfer rate or concentration in two-dimensional compartment) as well as a volume (for a three-dimensional compartment) [73].

In our provenance example of Lee et al. (2003), QM1 contains a qualitative model in the form of a chemical reaction network diagram which can be directly accessed via a DOI. It represents reactions of the species Wnt, Dsh, GSK3, Axin, APC, β-catenin, and TCF within a cell extract.

Simulation Model [SM]:

The simulation model is the actual mathematical or computational model [74] that can be executed by a suitable tool. In most cases of our domain, the simulation model contains equations (for ODE/PDE systems) or, in some cases, reaction rules (for rule-based systems). An integral part of these quantitative simulation models are the parameter values as well as the initial condition. The simulation model could also be described in another form (e.g., in a quantitative process algebra [75, 76] or with a combination of multiple formalisms [77]). Formal approaches to describe a system through qualitative models (e.g., Boolean models [78] or Petri nets [79]) come with their own means of analysis and are assigned to the Simulation Model entity as they are executable models. Usually, a parameter table complements the description of the simulation model and gives information about the parameter values and their origin.

A new Simulation Model entity is created whenever the reaction network changes or after a simulation model has been calibrated, which typically means that the set of parameter values and the initial condition have been (re-)defined. A validation activity (by itself) does not alter the simulation model, although a failed validation activity is likely to induce a change of the simulation model (see, for instance, [56]).

Again, we are relying on a reference of the simulation model for accessing it. It should be a link to the simulation model in Biomodels or a DOI to the description of the simulation model. Ideally, it is presented in a structured and widely accepted format such as SBML [80] or CellML [81].

As for the provenance example, the calibrated simulation model of Lee et al. (2003), SM2, can be found in BioModels.

Simulation Experiment [SE]:

The simulation experiment is an execution of the simulation model. Ideally, it can be linked to a complete experiment specification (e.g., as a SED-ML [82] or SESSL [83] file or simply as the execution code in a general purpose programming language) and to documentation in a standard format that applies reporting guidelines such as MIASE for simulation experiments [8]. Different simulation experiments might be used for the analysis, calibration, and validation of a simulation model.

To further structure the set of applied simulation experiments, we distinguish simulation experiments by whether they are used for optimization, sensitivity analysis, perturbation, parameter scan, steady-state analysis, or time course analysis. This list is neither complete nor are the categories disjoint, and, given a different set of simulation studies, they will likely be subject to renaming, extension, and refinement.

We have defined optimization experiments to be all experiments where an implicit or explicit objective function is used. This includes parameter estimation as well as manual parameter fitting experiments. If these succeed, the Calibrating Simulation Model activity will produce a new (calibrated) Simulation Model. In a sensitivity analysis, more than one parameter value is changed and some kind of sensitivity coefficient is calculated. We have interpreted experiments where the value of one (or more) parameter is changed to another (just one) value, for example, to mimic a knock-out experiment as perturbations. Parameter scans include the variation of at least one parameter value within a given range. A steady-state analysis is aimed at identifying the steady state of a system. We refer to time course analysis to be the analysis of trajectories without varying parameter values.

Eventually, an ontology about the various experiment types and analysis methods and their use in simulation studies will be crucial as simulation experiments play a central role in the provenance of simulation models. This would also help to exploit the provenance information effectively, for example, for automatically generating simulation experiments [84].

In the case of Lee et al. (2003), different simulation experiments have been executed. For example, SE1 contains a parameter scan in order to validate the simulation model. However, no further details are given in the paper, therefore, no reference could be included in the entity (the reference is “not available”).

Note that we have not included a Wet-lab Experiment entity. Our focus is on the result of the Wet-lab Experiment (i.e., the Wet-lab Data) and its role within the simulation study (e.g., being used in a Building Simulation Model, Calibrating Simulation Model, or Validating Simulation Model activity).

Simulation Data [SD] and Wet-lab Data [WD]:

Data is the result of an experiment. In our case, it can either stem from wet-lab or simulation experiments. It includes a reference to a plot or table or, ideally, to a database containing the raw data. Because simulation data is generated by a simulation experiment, a link needs to be established. In the case of a simulation experiment that serves the role of validation, information about the success of a validation facilitates the interpretation of the simulation model and further activities based on the simulation model. As no independent Wet-lab Experiment entity is supported, details about the wet-lab experiment can be summarized in the description of the Wet-lab Data or by referencing, for example, a research protocol on Protocols.io [85]. The type of the wet-lab experiment (in vitro or in vivo) as well as the used organism and organ/ tissue/ cell line should be recorded.

In the provenance example, Lee et al. (2003) have executed in vitro wet-lab experiments with an egg extract of Xenopus and have shown in WD1 that the “turnover of GSK3β, Dsh, and TCF is relatively slow” [28]. The data from this wet-lab experiment is not shown in the publication. The simulation data SD2 contains the results of the successful validation of the simulation model SM2. The simulation data is presented in Figure 2 of their publication. The way the provenance graph and the metadata of SD2 is visualized in WebProv can be seen in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Screenshot of WebProv.

This screenshot shows the provenance graph of the study by Lee et al. (2003) [28] with additional entities from three other studies [64–66], which are automatically colored differently. The node SD2 has been clicked on, which opens a box on the right with the stored and editable metadata.

https://doi.org/10.1371/journal.pcbi.1009227.g004

Provenance activities and relations.

The provenance graph is formed by explicitly relating entities and activities. This is done by declaring which entities are being used or which entities are being generated by which activities. We currently distinguish four activities: building, calibrating, validating, and analyzing the simulation model.

Products of activities (i.e., entities) are connected to these activities via the relation wasGeneratedBy. For example, Simulation Experiments or Simulation Data may be the result of all but the Building Simulation Model activity. Activities are connected to entities via the relation used. For example, the Calibrating Simulation Model activity may use the Simulation Model as the object to be calibrated, some Simulation Data or Wet-lab Data for calibration, and Requirements to confirm the calibration results. All connections that we currently distinguish are shown in Table 2.

It should be noted that provenance activities in simulation studies can be defined at various levels of granularity. We have opted for a rather coarse-grained approach identifying only crucial activities of a simulation study without explicitly denoting how an activity has used a specific entity. Thus, we aggregate activities as much as possible and leave out intermediate steps, focusing on the entities and not on the activities. From the moment provenance information is recorded automatically during the course of a simulation study, a higher level of detail could be achieved and an abstraction-based filter could be applied to zoom out to reach our granularity [2].

Building Simulation Model [BSM]:

The Building Simulation Model activity, also called model derivation [86], can use all entity types as any entity described above can contribute to the model building process, but it needs to have at least one link to a Research Question or Simulation Model. The only result of the building simulation activity is a Simulation Model entity. Not every update of a simulation model within a simulation study will be reflected in the provenance graph—only those changes to the model that are considered essential by the authors.

In our provenance graph of the study of Lee et al. (2003), two Building Simulation Model activities are shown. BSM1 is using wet-lab data, the research question, the qualitative model, a requirement and assumptions to develop a “provisional reference state model” [28], which forms the not yet calibrated simulation model in the study. The Building Simulation Model activity BSM2 extends the simulation model SM2.

Calibrating Simulation Model [CSM]:

The calibration of a simulation model is used to determine parameter values. Sometimes, switching parts of a model on or off (e.g., individual rules or model components) or choosing between entire models can also be interpreted as a discrete parameter value to be determined using methods of model selection [87]. This activity uses a Simulation Model and typically needs reference data (Wet-lab Data or Simulation Data) for the parameter estimation procedure and produces a specification or documentation of a Simulation Experiment as well as a Simulation Data entity. If the calibration is successful, the result of this activity will always be a (calibrated) Simulation Model. Ideally, it also takes an explicit requirement into account, which, in some cases, if formally defined, can also be used for calibrating the simulation model [88, 89]. It may also use an Assumption.

In the case of the activity CSM1 from Lee et al. (2003), several wet-lab data from their own experiments (WD1–WD5) as well as from Salic et al. (2000) [66] (WD1–WD3) are used during the calibration of the model SM1 which produces a Simulation Experiment SE1, the corresponding Simulation Data SD1 as well as the calibrated Simulation Model SM2.

Validating Simulation Model [VSM]:

The validation of a simulation model is used to test its validity (with regard to some requirements). Unlike calibration activities, here, the result is typically a binary answer, yes or no, which may be determined based on a specific distance measure and error threshold. The activity uses a Simulation Model and traditionally relies on reference data (Wet-lab Data or Simulation Data) that has not been used for calibration. For example, the Simulation Data from other studies may be used for the intercomparison of simulation models when performing an equal simulation experiment. Additionally, the required behavior can be formally specified in a Requirement (e.g., in a temporal logic) and automatically be checked [90, 91]. The Validating Simulation Model activity may also use an Assumption. It produces at least one entity of type Simulation Experiment as well as the corresponding Simulation Data entity. The Simulation Data entity of validation experiments stores the information whether the validation has been successful or not.

In our provenance example of Lee et al. (2003), the simulation model SM2 is validated in the activity VSM1 by comparing it with their own in vitro measurements shown in WD6 and WD7. Neither a distance measure nor an explicit requirement are mentioned.

Analyzing Simulation Model [ASM]:

Similar to validation and calibration experiments, this activity provides insights into the simulation model and thus also into the system under study. The activity uses a Simulation Model and creates a Simulation Experiment as well as resulting Simulation Data. The use of Assumptions, Simulation Data, or Wet-lab Data is optional and might give an indication about the purpose of an analysis. The Analyzing Simulation Model activity aggregates all simulation experiments that are not explicitly aimed at calibration and validation.

Lee et al. (2003) analyze both the calibrated simulation model SM2 as well as the extended simulation model SM3 by applying parameter scans, perturbations, and sensitivity analyses which results in the Simulation Experiment SE3–SE11 and Simulation Data SD3–SD11.

Extensibility and applicability of the approach.

All of these entities, activities and relations show major steps of the development of a simulation model and, as we will see in the following section, help to interrelate different simulation studies. Still, PROV-DM would allow even more details. We have not yet considered the type agent from PROV-DM in our approach because this information appeared less relevant in the analyzed simulation studies. In the future, the provenance information could include the name of the agent an entity is attributed to, the agent an activity is associated with, or the name of the agent another agent acted on behalf of [19]. This would be of particular relevance if models are not only validated but also accredited, which typically involves a different group of people other than those who have developed the simulation model [92].

We have also not included the direct connection between two activities or two entities, such as the possibility to have a model being derived from another model. Thus, we have not included the following relations: a) WasInformedBy, which relates an activity to another one and b) WasDerivedFrom, which describes a direct transformation (update) of an entity into a new one. However, these relations can partly be inferred via the existing relations. For example, a simulation model that has been generated by a Building Simulation Model activity that used another simulation model indicates that the former has been derived from the latter. Additionally, a validation activity that failed and that is followed by a Building Simulation Model activity obviously holds some information for the latter.

In our experience, it is best to capture provenance information manually or (semi-)automatically during the modeling (and simulation) process. This could be done, for example, within an artifact-based workflow system [2]. However, this would rely on a fixed life cycle definition (i.e., constraints regarding the allowed activities). Other approaches are based on automatically analyzing scripts, either with [93] or without [94] the help of user annotations. The latter has potential for a fully automatic and transparent provenance capture, however, it is difficult to implement, highly system-dependent, and only accounts for provenance information contained explicitly in the scripts, thus leaving out important information, such as research question, assumptions, or qualitative models. WebProv, on the contrary, is a standalone tool that works system-independently. Although it requires user input or a valid JSON input file, it also provides great flexibility regarding the information to be captured.

Provenance of individual Wnt simulation models.

It is important to remember that we have manually collected all provenance information (entities, activities and relations), as described in the Materials and methods section. Collecting this information based on publications only is a demanding task and requires some interpretation, as natural language descriptions tend to be ambiguous. Also, the nonlinearity of the text—it is not a lab protocol after all—makes it hard to identify the relations between the entities and activities as well as the order of their execution. This would likely hamper an effective use of text mining or machine learning methods to complement or replace the manual work. Therefore, provenance information should be collected during the simulation study and ideally without an intervention of the modeler.

The Research Question was usually repeated within the abstract and throughout the introduction and discussion sections. Sometimes, there was more than one research question to be answered. In this case, we have combined these into a single entity.

Many Assumptions were introduced by the word “assume” or its derivatives. Other expressions such as “hypothesis”, “is believed”, “consider”, “approximate”, “simplify”, “suggest”, “suppose”, or “propose” were also used by the authors to mark an assumption. However, not every sentence containing one of these words was identified to be an assumption of the simulation model. Occasionally, there were also assumptions which did not use one of the key words from above. Furthermore, two out of 19 studies did not explicitly state assumptions ([29, 53]). Generally, identifying assumptions involves many uncertainties. On the one hand, the authors might not have stated all assumptions made during the derivation of the simulation model. On the other hand, we could have easily missed an assumption or interpreted statements as assumptions that were not meant as such by the authors. Consequently, the assumptions might look different if the authors had defined them themselves.

In order to further analyze the assumptions, we categorized them using the Systems Biology Ontology (SBO) [68]. However, the assumptions could not always be unambiguously matched to an SBO vocabulary and some assumptions dealing with biological mechanisms are not covered by SBO. For example, an autocrine signaling assumed by Mazemondet et al. (2012) [46] cannot be expressed by SBO. Some assumptions also include more than one detail which is reflected by multiple SBO categories per assumption. In this case, the assumption entity is duplicated and every assumption entity will receive its own category. The categorization of 106 collected assumptions shows that the three most used categories of assumptions deal with kinetic constants (13 times), transport (9 times), and equivalence (8 times). The result of the categorization can be found in S1 Table.

In many cases, Requirements were not given explicitly in a formal way or even as textual descriptions. We could only identify Requirements in the publications of Lee et al. (2003) [28], Wang et al. (2013) [48], Chen et al. (2014) [49], and Haack et al. (2020) [56]. The lack of Requirements was especially obvious when calibration or validation experiments were carried out without explicitly explaining the objective function.

In the surveyed publications, it was common to include a reaction scheme of the simulation model, which we referred to in the Qualitative Model entities. When recording all species, we disregarded di- or multimeric compounds established by monomers already mentioned. We also ignored different states of the species (e.g., phosphorylation states). In all provenance graphs but the one by Mirams et al. (2010) [41], at least one Qualitative Model was used by a Building Simulation Model activity to produce the first Simulation Model. Mirams et al. (2010) have directly worked with the simulation model SM2 from Lee et al. (2003).

The Simulation Models were either part of the manuscript or, more often, part of the supporting material. In 13 out of 19 cases, it was a system of ordinary differential equations. There were two simulation studies using PDEs and four using a rule-based approach (see Table 1). Although the Wnt signaling pathway has also been used to illustrate features of rule-based modeling [95, 96], only few simulation models have been developed based on a rule-based approach. The reason for this might be partly because support for thorough experimentation with rule-based models including calibration and validation has only become available during the last decade [97–99].

We categorized all 145 Simulation Experiments that we found depending on which experiment type they served (see UML class diagram shown in Fig 2). The results of the categorization are shown in Fig 5 and the details in S2 Table. Most Simulation Experiments were parameter scans, followed by time course analyses and perturbations. None of the 19 simulation studies have used steady-state analysis alone without applying another type of experiment at the same time, which we then recorded because it was more specific. The detection of steady states is typically part of an optimization, parameter scan, perturbation, and sensitivity analysis, because steady-state values are often the starting and end point of each simulation and are used for calculations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Categories of the simulation experiments conducted within the analyzed 19 Wnt signaling studies.

All of these simulation experiments have been categorized. Most simulation experiments were parameter scans.

https://doi.org/10.1371/journal.pcbi.1009227.g005

Sometimes, simulation or wet-lab experiments have been conducted, however, the corresponding Simulation Data or Wet-lab Data is not shown. Instead, they are briefly described, and thus are without references in the provenance graphs. Authors often refer to this by stating “data not shown”. For example, Lee et al. (2003) state that unpublished measurements showed that the “turnover of GSK3β, Dsh, and TCF is relatively slow” [28]. Usually, both Simulation Data and Wet-lab Data are shown in figures in the studies or published in tables or figures as part of the supplemental material. In recent years, more and more journals, such as PLOS Computational Biology, have been recommending (but not requiring) to adhere to checklists of the FAIRsharing [100] portal when reporting data, with FAIR standing for: findable, accessible, interoperable and reusable [101].

In the case of Simulation Data, the focus lies on FAIR simulation models and experiments as it should be possible to easily regenerate the data. This could be achieved, for example, by publishing a COMBINE archive [61], which is a “single file that aggregates all data files and information necessary to reproduce a simulation study in computational biology” [102]. However, the publications we have analyzed date back up to 17 years, so most data has not been published in a FAIR way.

When looking at Wet-lab Data, six out of 19 publications recorded their own wet-lab data [28, 31, 32, 45, 51, 57]. All other simulation studies either used wet-lab data from other publications or did not explicitly refer to wet-lab data at all [29, 30, 33, 34, 39, 41, 53]. The latter could only be done because the authors relied on other Simulation Models and their respective parameter and initial values. Interestingly, the wet-lab data obtained by the 19 Wnt signaling studies and the other studies which were used in the 19 studies stemmed from different organisms and cell lines. Besides human and murine cell lines (each 11 studies), xenopus, rat, hamster, and kangaroo rat were used as a model organisms. The experiments included, among others, (tumor) cell lines from the kidney (BHK, HEK293, PtK2), bone (MG-63), cervix (HeLa), brain (neural progenitor cells), and fibroblasts (L cells, NIH/3T3). All different cell lines directly used in the studies are presented by the colored rectangles in Fig 6. The cell lines and organisms that were included in the simulation studies are shown in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Provenance graph of all Wnt/β-catenin simulation studies considered here (black outlines) as well as additionally used studies (gray outlines).

The colors indicate the cell lines used in wet-lab experiments of that study (see legend below graph). Gray boxes represent pure Wnt/β-catenin simulation studies without acquiring wet-lab data. White boxes display publications used by some of the Wnt/β-catenin simulation studies that are either text books or simulation studies without published wet-lab data. The figure was created using the R package DiagrammeR [103]. The references to the additionally used studies are found in S1 Appendix.

https://doi.org/10.1371/journal.pcbi.1009227.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Cell lines and organisms used building the simulation models.

https://doi.org/10.1371/journal.pcbi.1009227.t003

When looking at the CSM and VSM activities, we detect that only a few simulation models were both calibrated and validated based on wet-lab data, namely those of Lee et al. (2003) [28], Kogan et al. (2012) [45], and Haack et al. (2015) [51]. Some authors just calibrated their simulation models ([33, 46, 48, 49, 53, 57]) and in two studies ([30, 56]) they were only validated (having assumed plausible ranges for parameter values). The simulations models developed in [29, 32, 34–36, 39, 41] had neither been calibrated nor validated explicitly, because they used parameter values from other studies. Sick et al. (2006) [31] used arbitrary parameter values in their simulation models.

Some studies develop a story line where a simulation model has been successively refined, extended, or composed ([28, 33, 34, 36, 41, 46, 48, 49, 51, 56]). These studies are characterized by a Simulation Model being used by a Building Simulation Model activity, both nodes of the same study.

Some simulation studies resulted in the development of multiple simulation models that are neither extensions nor compositions but rather form a revision or alternative to other simulation models developed in the same study: [33, 39, 41, 46, 48, 49]. This can be seen in the provenance graph of a single study when the last simulation model is not connected by a directed path to other simulation models of that study or when the simulation models are part of disjoint branches of the provenance graph. For example, in Mazemondet et al. (2012) [46], the core model of the Wnt/β-catenin signaling pathway has been calibrated with wet-lab data from Lee et al. (2003) [28], but this calibrated simulation model was not used and instead, a new calibration with wet-lab data from another study took place. Two simulation studies ([35, 41]) show disconnected graphs. This shows that these researchers considered, built, and analyzed multiple simulation models independently of each other.

Beyond individuals: A family of Wnt simulation models.

Whereas before we have looked at the properties of individual simulation studies, we are now going to investigate the interrelations between the 19 Wnt/β-catenin signaling simulation models. We will identify features that transform a set of simulation models into a family.

Fig 6 shows an overview of all simulation studies considered in our research by zooming out of the individual provenance graphs. All but two simulation models are (indirectly) connected to the model proposed by Lee et al. (2003) [28], which was the first validated simulation model of the canonical Wnt signaling pathway. This shows that models are usually not built independently from one another, but are often extensions or revisions of formerly published models or use the parts of the reactions or parameter values. The two exceptions not linked to Lee et al. (2003) [28] are the simulation models by Sick et al. (2006) [31] and by Rodríguez-González et al. (2007) [33]. Sick et al. (2006) use a reaction-diffusion system (Gierer-Meinhardt equations [104]) to model the interplay between Wnt and its antagonist Dkk with just two equations. Rodríguez-González et al. (2007) consider Wnt and Axin—two key players of the Wnt/β-catenin signaling pathway—but only in the context of the Notch signaling pathway and thus the model only contains a fragment of the canonical Wnt pathway.

We have also included the cell lines/tissue used for wet-lab experiments within each study. As already seen in Table 3, we observe that different simulation studies use data or models obtained using other cell lines. This may be valid as the Wnt/β-catenin signaling pathway is evolutionarily conserved [24], which means that data can be shared. Still, care must always be taken when using, for instance, parameter values determined with one cell line in a study that uses another cell line.

When looking at the same graph using a circular layout, we observe four clusters of two or more studies, as shown in S1 Fig. We have also colored the studies according to additional pathways they include and observe that the clusters separate the studies depending on these additional cellular mechanisms. The central cluster includes the Wnt model by Lee et al. (2003) [28] as well as the studies [30, 34, 35, 41]. A second cluster forms around the simulation studies [45, 46, 51, 56, 57] and either includes the same wet-lab data from [105, 106], the Cell cycle [46] or ROS [51, 57]. A third cluster includes the pathways of Notch [33, 48] and Notch + MAPK/ERK [36]. Even though the algorithms locate [29] in the same cluster, it is content-wise rather part of the central cluster. A forth cluster forms around studies that include MAPK/ERK [32] or MAPK/ERK + PI3K/Akt [53]. All other models are not part of a cluster and are either completely disconnected from the other studies [31] or include E-cadherin and the cell cycle [39] or just E-cadherin [49].

When comparing Fig 6 and S1 Fig, we find that the wet-lab data from just three studies, namely from [105–107], have been reused by simulation studies. On the one hand, this is surprising as wet-lab data can be reused for parameter estimation or model validation. On the other hand, when authors use parts or entire simulation models published by others, they do not necessarily recite the references that were used for obtaining the parameter and initial values that come with the model. Thus, a direct connection from the new simulation model to the wet-lab data used by another simulation study is not made.