Countering reproducibility issues in mathematical models with software engineering techniques: A case study using a one-dimensional mathematical model of the atrioventricular node

One should assume that in silico experiments in systems biology are less susceptible to reproducibility issues than their wet-lab counterparts, because they are free from natural biological variations and their environment can be fully controlled. However, recent studies show that only half of the published mathematical models of biological systems can be reproduced without substantial effort. In this article we examine the potential causes for failed or cumbersome reproductions in a case study of a one-dimensional mathematical model of the atrioventricular node, which took us four months to reproduce. The model demonstrates that even otherwise rigorous studies can be hard to reproduce due to missing information, errors in equations and parameters, a lack in available data files, non-executable code, missing or incomplete experiment protocols, and missing rationales behind equations. Many of these issues seem similar to problems that have been solved in software engineering using techniques such as unit testing, regression tests, continuous integration, version control, archival services, and a thorough modular design with extensive documentation. Applying these techniques, we reimplement the examined model using the modeling language Modelica. The resulting workflow is independent of the model and can be translated to SBML, CellML, and other languages. It guarantees methods reproducibility by executing automated tests in a virtual machine on a server that is physically separated from the development environment. Additionally, it facilitates results reproducibility, because the model is more understandable and because the complete model code, experiment protocols, and simulation data are published and can be accessed in the exact version that was used in this article. We found the additional design and documentation effort well justified, even just considering the immediate benefits during development such as easier and faster debugging, increased understandability of equations, and a reduced requirement for looking up details from the literature.

• As suggested by reviewer #1, we added a paragraph discussing interoperability and the related advantages of SBML/CellML in the discussion. • We searched through the whole article for statements that painted the work of Inada et al. in an overly negative light and corrected them, eliminating judgmental language and highlighting the overall importance of the Inada model. We especially tried to clarify that we believe the source of the reproducibility issues is the complexity of the model and not a lack of diligence of the authors or the reviewers. • Since InaMo 1.4.3, we now include an export of the main model as Functional Mock-up Unit (FMU), which should increase the interoperability of InaMo at least within the Modelica domain. • We have switched from Travis CI to GitHub actions, because Travis CI is no longer free for public open source repositories. We therefore replaced all mentions of Travis CI as well as the code examples for the CI/CD scripts. • Some statements in the manuscript did not reflect the fact that we performed a manual reconstruction of the data of all reference plots shown in the supplement. We updated the affected sentences accordingly. • While going through the article, we found a few issues with spelling, grammar, and references. In particular, we updated the date on all web resources from 2020 to 2021, replaced bioRxiv references with published versions, and replaced the reference to Juty 2013 with Courtot 2011, because we believe that this is a better reference for the SBO.
Our detailed responses are listed in the following along with the original comments.
Kind Regards, Christopher Schölzel, Valeria Blesius, Gernot Ernst, Alexander Goesmann, and Andreas Dominik Reviewer #1 Comment 1.1 "My only criticism of the study would be that the authors' use of Modelica comes at the expense of more exchangeable formats such as CellML/SBML. The Modelica code provided with this submission is very complete, but the CellML and SBML formats are quite a bit more portable, and can be imported and exported by many tools, and the current study does not appear to have an accompanying CellML/SBML model. The authors argue that CellML/SBML unsuited to first-hand model authoring due to their lack of human readability and problematic interaction with version control systems. For this particular study, I agree with this assessment. I believe that providing CellML/SBML code for this model would be against the DRY principle and create synchronization issues and hence barriers to robust reproducibility (and certainly, it would not be difficult to derive a CellML/SBML version of this model given the wealth of tests and documentation provided by the authors). However, I would view it as a regression if exchangeable standards such as CellML/SBML were eventually replaced with less transferable counterparts. I think the authors can address these issues as discussion items, rather than make any implementation changes. I think an ideal solution would be to pursue an authoring medium, perhaps a meta-language, that retains the clarity and pragmatism of Modelica (for example) but can be interconverted to/from standards like CellML/SBML. Perhaps the authors can create a new paragraph in the discussion section that acknowledges the model exchangeability issue, considers the pros and cons of various approaches, and makes recommendations regarding how the field could approach a solution to this issue in the future. The authors already mention some of these issues around line 900. This section could be further expanded to offer recommendations and potential solutions for practical modeling languages and workflows that (I hope) attain all of the authors' criteria as well as exchangeability and annotation support that standards were designed to address."

Response:
We fully agree with the reviewer, and thank them for pointing out this oversight. While we do favor Modelica for model design, CellML/SBML have an undeniable advantage as exchange formats. In fact, we would be happy to provide a separate SBML or CellML version of the model, if there was a way to generate it automatically from Modelica code. As far as we are aware, such a conversion is currently only possible in the opposite direction with SBML2Modelica. This provides some interoperability between the two domains, but is insufficient to make our implementation directly accessible within an SBML-or CellML-based tool. The following approaches could remedy this in future projects: -SBML/CellML tools could be extended to support the Functional Mock-up Interface (FMI), which is the designated exchange format in the Modelica domain. The only modeling tool we are aware of that supports both FMI and SBML is MATLAB with the SimBiology and FMI toolboxes. To facilitate this form of interoperability, we now also provide the three main model as Functional Mock-up Unit (FMU). The downside is that FMUs are opaque boxes, which do not allow a partial reproduction of results that requires a structural modification of the model.
-Like the reviewer said, a standard, human-readable text representation of CellML/SBML models with sufficient support for modularity would allow to apply the same techniques presented in this article to CellML/SBML models. Antimony and CellML Text come to mind as possible candidates.
-A full automatic translation of Modelica to SBML/CellML would probably be impossible, because SBML/CellML does not have full support for DAEs, discrete variables, and the object-oriented modeling style of Modelica. However, it could be achieved for a well-defined subset of the language. Upon compilation, Modelica models are "flattened", removing the hierarchical structure and collecting all variables and equations in one single file. Starting from this representation, a translation to SBML/CellML seems achievable, yet still quite complicated due to discrete events, Kirchhoff-like algebraic equations, and custom functions with imperative syntax. As an additional downside, the "flat" Modelica code loses most of the benefits regarding understandability and modularity of the model.
As suggested by the reviewer, we added a paragraph in section 4.3 that discusses this issue.

Editor Comment 2.1
"My impression of the manuscript is that it is thorough, meticulouslyresearched and of use to the field. However, the many expressions of frustration with the Inada model should be minimized throughout the text."

Response:
We agree with the editor. We already tried to avoid emotional or judgmental language, but realized that we can do a better job to keep a factual tone throughout the text. In particular, we noticed that the reader can get the impression that we blame the authors and/or the reviewers of the Inada et al. study for the reproducibility issues. However, while we cannot deny that we felt frustration throughout the development, we believe that such issues are a natural result of high model complexity and that they can only be solved reliably with automated tool support. We tried to clarify this throughout the manuscript and also highlight that we do believe that Inada et al. were interested in making their methods as transparent as possible.
In a way this makes the Inada model even more interesting as an example, because it shows that even high quality research is subject to the issues that we encountered.