Dear Matthias,

It was not possible for us to benchmark all systems biology chemical reaction network simulation packages. Those we selected were influenced by a variety of factors, including popularity via citations and other statistics, being a vendor-based solution for their platform (i.e. Matlab SimBiology), and apparent development activity when we initiated the benchmark studies (2022). (At that time the documentation we found via Google and the libroadrunner.org website redirected us to web.archive.org, giving us the impression the project was no longer under active development.)

Given your comment above, and Herbert Sauro’s earlier email that stated similar points about libRoadRunner, we recently benchmarked RoadRunner on our test set. We used the Julia interface (https://github.com/SunnyX...), and also asked the RoadRunner developers for advice in setting it up. We were unable to get RoadRunner to work on our HPC, and as such, we ran the benchmarks on my local machine (and also re-ran the Catalyst benchmarks on the same machine for comparison).

Unfortunately, we were unable to reliably benchmark RoadRunner on the two larger models (fceri_gamma2 and BCR), experiencing issues that included failures to load the models and very long run times. Here we provide the results of the benchmarks for those cases we could successfully run, where we in each case re-ran the benchmarks to confirm they were reproducible:

The benchmarks are available via the following link: https://gist.github.com/T...

The code for performing these benchmarks can be found as the “RoadRunnerBenchmarks” branch of the Github repository for this paper. For Catalyst, we used the most performant options (as found in Figure 3 of the paper, and described in detail in Table 3). For RoadRunner, we used the default options, as no others were either described in the documentation (https://sunnyxu.github.io...), or in the scripts and correspondence we received when setting these up.

We are content with the extent of these additional benchmarks, and plan to focus on other development priorities going forward. However, if you would like to perform additional studies we are happy to assist with any questions on using Catalyst.

How we benchmark all tools, including COPASI, is described in the paper, with the scripts used available in the online repo. We also specify each tool’s version (the latest at the times the benchmarks were carried out). The paper describes the various benchmark options and configurations we examined.

We agree that correctness of implementations is important. The link you provide focuses on tests regarding SBML models, a file format for systems biology models. Catalyst SBML support is provided via SBMLToolkit.jl, a separate package from Catalyst, with development led by others. While we have previously added features to Catalyst to enable better SBMLToolkit coverage of the SBML standard, any SBML-related benchmarks and coverage matters would be better discussed with the SBMLToolkit developers via an issue on their Github repo. We are happy to add features to Catalyst to better enable SBML functionality as appropriate, and would be enthusiastic to work with anyone affiliated with the official SBML libraries if there is interest in integrating with Catalyst.

Catalyst functionality is validated by our own set of unit and CI tests, and tests of the underlying solver libraries that Catalyst leverages. These are publicly available for inspection, and run via CI before any code update is merged. We are happy to add additional tests should there be Catalyst components that are not adequately covered by our current CI, please feel free to open a Github issue with suggestions!

Best wishes,
Torkel Loman