Overview

We present a methodology for the robust calibration and validation of dynamic models of microbial communities described by nonlinear Ordinary Differential Equations (ODEs). Dynamic models of microbial communities, such as generalized Lotka-Volterra (gLV) or consumer-resource models, possess distinctive features compared to many other models in computational systems biology [28–30]. While challenges like nonlinearity and high dimensionality are common in systems biology, their specific combination and context in microbial ecology, plus the presence of other specific challenges, create a uniquely difficult parameter estimation problem that motivates a tailored computational workflow. The primary challenges are:

1. Strong Nonlinearity and Highly Nonconvex Landscapes. Microbial community models explicitly encode complex interaction networks through nonlinear terms. This inherent nonlinearity gives rise to complex, multimodal dynamics and, consequently, a highly nonconvex objective function landscape. This makes optimization susceptible to local optima, hindering the identification of globally optimal solutions.
2. Dense Model Structures and Data-Driven Identifiability Issues. Unlike many systems biology models with defined topologies (e.g. biochemical pathways), certain types of models like gLV often assume a dense, all-to-all interaction structure. This leads to a quadratic () increase in parameters with the number of species (N), resulting in extremely high-dimensional systems. This structural complexity is compounded by severe identifiability challenges rooted in the symmetries of the interactions and the lack of sufficient informative data (given the large number of parameters). Many parameters, especially interaction coefficients, can be structurally or practically non-identifiable from typical community-level data. Data often consist of relative abundances or aggregated signals, which complicates the unique resolution of absolute parameter values. Observability is limited since some species or resources may be unmeasured or measured only coarsely.
3. Phenomenological Nature and Proneness to Overfitting. Compared to more mechanistic frameworks (e.g., metabolic models constrained by stoichiometry), gLV and similar models are largely phenomenological. The large number of parameters often lacks strong theoretical or prior constraints. This high degree of freedom, combined with what is often sparse time-series data, creates a significant risk of overfitting, where the model captures noise in the training data but fails to generalize.
4. Potential for Numerical Instability. Due to their nonlinear structure, these models (particularly gLV formulations) can exhibit solutions with finite-time blow-up (unbounded growth). This causes numerical instability during parameter estimation, as many candidate parameter sets lead to ODE solver failures. Calibration algorithms must therefore incorporate strategies to handle or constrain the search to dynamically stable regions.

Overall, although these distinctive characteristics and challenges are not strictly exclusive to microbial community models, the confluence of a dense parameter structure, the potential for overfitting and numerical instability, and a lack of mechanistic constraints does present a distinct and severe challenge. To the best of our knowledge, the literature lacks a comprehensive framework that systematically addresses this specific combination of issues, which is the gap our work aims to fill. As previously indicated, only recently have a few studies [24–26] begun addressing some of these issues (e.g., identifiability analysis for selected models). Thus, a key contribution of our paper is to introduce and demonstrate the latest methods for identifiability analysis and robust model calibration specifically in the context of microbial community modeling, where these challenges are particularly pronounced.

Our integrated computational workflow has been developed to address these issues and consists of three sequential phases, each subdivided into modular subtasks:

• Phase 1, pre-estimation preparation, begins with structural identifiability analysis (SIA) to determine whether model parameters are theoretically distinguishable. If simplification is required, users can remove non-identifiable parameters, reparameterize equations to reduce complexity, or explore alternative input-output mappings by adjusting perturbed or measured variables.
• Phase 2, robust parameter estimation, employs global optimization techniques to avoid local minima, paired with efficient adaptive ODE solvers and mechanisms to address numerical instabilities and overfitting. After estimation, practical identifiability analysis (PIA) is used to assess whether model parameters can be reliably estimated from the available data, considering measurement noise and experimental constraints. When needed, it can be followed by dynamic stability analysis via Jacobian eigenvalue evaluations to assess system robustness.
• Phase 3, predictive power analysis, checks possible over-fitting and its consequences. It also evaluates predictive power and generalizability by comparing simulations against experimental data and additional cross-validation tests with held-out datasets. The results can help users to refine the model if needed. Refinement options include simplifying the model structure, redesigning experiments to improve data quality, or collecting additional data to resolve ambiguities.

This workflow, implemented via version-controlled scripts and interactive electronic notebooks, ensures reproducibility and transparency. By integrating theoretical rigor (e.g., identifiability checks, stability analysis) with computational robustness (e.g., global optimization, regularization), this pipeline minimizes risks such as over-fitting, non-identifiability, and unstable dynamics, yielding reliable models for real-world applications. The methodology is outlined in Fig 1. We describe the different phases in the following sections, with further technical details in S1 Text (Section 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Diagram illustrating the workflow for model calibration.

The figures correspond to a simple Generalized Lotka-Volterra model with 2 species in a competition scenario, specifically representing an in-vivo competitive mixture experiment with influenza strains in ferrets [27]. Further details about each phase of the workflow can be found in Methods.

https://doi.org/10.1371/journal.pcbi.1013204.g001

Pre-estimation preparation

The microbial communities under consideration are described by sets of non-linear Ordinary Differential Equations (ODEs) and the observation function as follows:

(1)

(2)

where , is the vector of state variables at time t, with initial conditions ; denotes the vector of model parameters within the feasible parameter space Θ, and corresponds to the external factors or inputs. regards the vector of n_o observables. We will regard as fully observed (FO) systems those for which all the state variables are amenable to experimentation and partially observed (PO), otherwise.

Structural identifiability analysis (SIA) seeks to determine which model parameters can be uniquely estimated from a given perfect data set, i.e., continuous and noise-free. It should be noted that in reality we never have perfect, noise-free data and measurements over infinite time. The question is whether parameter estimation is even theoretically possible given the model structure in Eqns (1)-(2). In general, we can distinguish:

• Global structural identifiability: parameters can be uniquely determined from the observations.
• Local structural identifiability: there is a finite set of parameter values that yield the same observations. In synthetic problems, parameters can be uniquely inferred near their nominal values, but multiple equivalent solutions may exist in the parameter space.
• Structural non-identifiability: there is an infinite set of parameter values that yield the same observations.

Structural identifiability is crucial for meaningful parameter estimation, as non-identifiable models yield unreliable and non-unique parameter values that may not reflect the true system properties. Inaccurate estimates of mechanistically significant parameters compromise the usefulness of the model to provide biological insights [21,31,32]. It also impacts model validation and interpretation, making it difficult to draw robust conclusions when parameters cannot be uniquely determined. Furthermore, identifying non-identifiable parameters informs new experiment design, guiding data collection and experimental strategies to improve parameter identifiability.

Several methods exist for studying SIA in nonlinear models [31,33], but they often rely on complex symbolic manipulations. While various software tools are available, they tend to be user-unfriendly and require significant expertise. Moreover, no single method applies universally to all models. These challenges create barriers that often lead to the neglect of this crucial step in model calibration.

After an initial screening based on our requirement to analyze structural global identifiability, as well as considerations of computational scalability, flexibility (including multi-experiment identifiability of both parameters and initial conditions), and previously reported performance [33], we selected the following tools for further evaluation:

• GenSSI2 [34] is a MATLAB-based tool that uses a generating series approach. It transforms the model equations into a system of polynomial equations on the parameters, analyzing their rank conditions to determine local identifiability and solving the system of equations to analyze global identifiability. However, its reliance on pure symbolic computations can make it computationally demanding for partially observed, highly non-linear systems.
• SIAN [35,36] is a Julia package that uses a randomized Monte-Carlo algorithm and an improved version of the Taylor series approach. It employs a differential algebra method that eliminates non-identifiable parameters through Gröbner basis computations.
• Structural Identifiability [37] is another Julia-based package that also uses differential algebra approach based on input-output equations. The method combines randomization and a differential elimination algorithm, and presents good scalability.

In principle, all these tools can handle both local and global identifiability, and can be applied to polynomial and rational ODE models. This selection offers a well-balanced combination of capabilities, performance, and suitability for the models under analysis. Each tool has unique strengths: GenSSI2 is effective for smaller to medium-sized models but can struggle with scalability due to its symbolic computation overhead, while SIAN and Structural Identifiability employ symbolic-numeric randomized algorithms and leverage Julia’s modern computational advantages for improved scalability and efficiency.

Robust parameter estimation

Dynamic model calibration is the process of adjusting the parameters to ensure accurate representation of the biological system’s behavior based on available data [38]. This process is typically framed as an optimization problem aimed at minimizing discrepancies between model predictions and observed data. Here we adopt a single-shooting approach, where the initial value problem described by the ODEs is solved for each evaluation of the cost function [39]. Other options are discussed elsewhere [40,41].

The choice of objective function plays a crucial role in quantifying this mismatch, as it directly impacts the accuracy and reliability of the calibration. The objective function encapsulates characteristics of the measurement process and incorporates prior knowledge. In frequentist approaches, it typically takes the form of the maximum (log)-likelihood function (details in S1 Text, Section 3.1). When standard deviations are known, this function follows a weighted least-squares form:

(3)

Where n_e, and regard the number of experiments, the number of observables in a given experiment e and the number of sampling times for the specific observables and experiment, respectively. represents the measured data at a given sampling time s for a specific observable o in the experiment e; denotes the model prediction for the sampling time ; and is the standard deviation of noise. The estimation problem is formulated as the minimization of one of these objective functions subject to differential and algebraic constraints (i.e., the ODEs describing the dynamics plus the observation function and bounds on the parameters, ).

In the above formulation, the parameter space is often high-dimensional and the problem is strongly non-convex, characterized by multiple local minima, flat regions, and possibly discontinuities caused by solver failures or finite-time blow-ups. These complexities make traditional gradient-based local optimization methods insufficient on their own, as they tend to get trapped in local minima and depend heavily on the initial guess.

To overcome these challenges, we adopt a global optimization strategy that can efficiently explore the parameter space and avoid premature convergence to suboptimal solutions. Specifically, we employ enhanced Scatter Search (eSS), a metaheuristic algorithm that integrates global search heuristics with periodic local refinement steps [42,43]. The key advantages of eSS include:

1. Balance of exploration and exploitation: eSS maintains a diverse population of candidate solutions to explore the parameter space broadly (exploration), while leveraging local optimization techniques to fine-tune promising solutions (exploitation). This balance helps in efficiently navigating complex landscapes typical of nonlinear ODE parameter estimation.
2. Robustness to nonconvexity and noise: the metaheuristic nature of eSS allows it to handle nonconvex and rugged objective functions, including those with discontinuities or noisy evaluations caused by numerical integration issues. This robustness is critical when the underlying ODE solver occasionally fails or produces invalid results (NaN, Inf).
3. Reliable convergence properties: compared to other popular stochastic methods such as genetic algorithms and other more recent evolutionary computation methods, eSS has demonstrated superior convergence speed and reliability in a range of parameter estimation benchmarks, due to its strategic combination of global and local search components [44,45]. Recent benchmarking studies have also illustrated its superiority over sophisticated multi-start gradient-based local methods [43].
4. Flexibility and adaptability: eSS is flexible enough to be combined with problem-specific constraints and can incorporate parallel evaluations, which is beneficial when solving computationally expensive ODE integrations [44,45].

Given these strengths, we found eSS particularly well-suited for the parameter estimation challenges typical of the class of problems considered here. It allows us to obtain high-quality parameter estimates with a higher probability of identifying global or near-global optima, ultimately improving the predictive accuracy and reliability of the fitted models, and justifiying its selection as the global optimization solver in our workflow.

Practical identifiability analysis (PIA).

To assess the practical identifiability of parameters, our workflow uses the Fisher Information Matrix (FIM), computing parameter confidence intervals derived from the Cramèr-Rao inequality, and examining the correlation matrix [19,20]. The process starts by constructing the FIM (eqn 4), which quantifies the amount of information the data is expected to provide about the parameters:

(4)

where E represents expected values and a value of the parameters, hopefully close to their real value (see below).

Computing the FIM involves calculating sensitivity matrices, which represent how parameter changes affect the model output, and incorporating assumptions about the measurement noise. The local parametric sensitivities for a specific experiment (e), observable (o), and sampling time (t_s) read as follows:

(5)

A key step is to evaluate the FIM’s properties: a singular or ill-conditioned FIM suggests practical non-identifiability, indicating that parameters or combinations of parameters cannot be reliably estimated from the data. The condition number and eigenvalues of the FIM can provide further insights into the degree of identifiability, with small eigenvalues or large condition numbers signaling potential issues.

Subsequently, confidence intervals for the parameters can be approximated using the Cramèr-Rao inequality, which states that the inverse of the FIM provides a lower bound on the variance of unbiased estimators:

(6)

where is the covariance matrix. By taking the square root of the diagonal elements of the inverse FIM, one obtains estimates of the standard errors for each parameter, which are then used to construct confidence intervals. Wide confidence intervals indicate poor practical identifiability, meaning that even with the assumed data quality, the parameters cannot be estimated with high precision. These wide intervals suggest that different parameter values, spanning a considerable range, could plausibly explain the observed data, thus hindering accurate parameter determination.

In practice, one typically evaluates the observed Fisher information at a point estimate (e.g., the maximum likelihood or nonlinear least squares estimator) and uses its inverse as a local approximation to the estimator’s covariance [20]. Although knowledge of the true parameter is not required, the Cramér–Rao lower bound is defined in terms of the information at the true value. Therefore under standard regularity conditions and for sufficiently large samples, the information at the estimate is a consistent proxy for the true information. This approximation is most reliable when the estimate is close to the truth, the parameter lies in the interior of an identifiable and smooth model, and the log-likelihood is approximately quadratic near the optimum. By contrast, pronounced nonlinearity, multimodality, boundary constraints, or nearly flat likelihood directions can render FIM-based confidence intervals and correlation measures inaccurate. In those situations, alternatives such as bootstrapping are recommended [46].

Finally, the correlation matrix, derived from the inverse of the FIM, offers insights into the interdependencies between parameter estimates. Off-diagonal elements of this matrix reveal the correlation coefficients between pairs of parameters. High correlation values, close to 1 or -1, suggest that these parameters are not independently identifiable, i.e., changes in one parameter must be compensated by changes in the correlated parameter to maintain a similar model fit to the data. This correlation implies redundancy in the parameterization and can guide model simplification or experimental redesign to improve identifiability. By collectively analyzing the FIM’s condition, the Cramèr-Rao based confidence intervals, and the parameter correlation matrix, one can gain a comprehensive understanding of the practical identifiability of parameters in ODE models, highlighting parameters that are poorly identifiable and suggesting potential remedies such as model refinement or improved experimental design.

Predictive power analysis

To assess the predictive power of a calibrated ODE model, a crucial next step is to investigate potential overfitting. Initially, this can be achieved by examining the residuals, which are the differences between the experimental data and the model predictions at the time points where data was collected. If the model is appropriately calibrated and not overfitting the training data, the residuals should exhibit a random distribution around zero, with no discernible patterns or trends. In a well-specified model with additive homoscedastic errors, in-sample residuals should behave like white noise, i.e. approximately zero-mean, uncorrelated, and of constant variance, with no systematic structure when plotted against time. In other words, assuming Gaussian errors in the data, residuals should also be approximately normal [20,47,48].

Visual inspection of residual plots, such as plotting residuals against time or predicted values, is helpful. Ideally, the residuals should resemble white noise, indicating that the model has captured the underlying signal and the remaining variation is attributable to random error rather than systematic model deficiencies. However, residual diagnostics alone do not reliably detect overfitting, which is better assessed via out-of-sample evaluation or cross-validation [20,48,49].

Thus, following residual analysis, a more robust evaluation of predictive power requires cross-validation. This involves partitioning the available data into training and validation sets. The model is then calibrated using only the training data to obtain parameter estimates. Subsequently, the predictive capability of the calibrated model is assessed by comparing its predictions against the validation data, which was not used during calibration. If possible, this process should be repeated multiple times for several sets of validation data. The predictive accuracy can be quantified using metrics like the root mean squared error (RMSE) or normalized root mean squared error (NRMSE) between the model predictions and the validation data across all folds. Consistently good predictive performance on the validation sets, comparable to the fit on the training data, indicates robust predictive power and absence of overfitting.

If the model demonstrates good predictive power and no signs of overfitting, the final step involves evaluating the mechanistic plausibility of the model fit. This is crucial, especially when dealing with models built upon mechanistic principles. The estimated parameter values should be examined for their magnitudes and signs in the context of their mechanistic interpretation. For instance, parameters representing rates of biological processes should have positive values and magnitudes that are biologically reasonable. Similarly, the signs of parameters governing interactions should align with the expected mechanistic relationships (e.g., a parameter representing inhibition should have a negative effect).

Mechanistic plausibility can go beyond general guidelines (i.e., verifying that parameter signs and magnitudes are consistent with expectations) to a domain-specific formulation that specifies the concrete criteria applicable to microbial community models. In this context, mechanistic plausibility checks can draw on established biochemical, microbiological and ecological principles [19,50–52]. For example, maximum specific growth rates (μ) should be positive and within realistic ranges (e.g., 0.1– for many bacteria), mortality or washout rates should be non-negative and generally smaller than growth rates, and interaction coefficients should have signs consistent with their ecological roles (positive for facilitation or cross-feeding, negative for competition or inhibition) and magnitudes that are biologically reasonable. Half-saturation constants (K_s) should be positive and within typical affinity ranges for the substrate (μM–mM), stoichiometric yields should not exceed theoretical metabolic limits, and parameter values should respect mass and energy balances. Moreover, feasible parameter sets should allow coexistence only under realistic nutrient inflow and removal rates, consistent with known microbial ecology constraints. Such domain-specific checks ensure that the model remains interpretable and biologically grounded even when statistical fit and predictive power are high.

Deviations from expected ranges or signs for mechanistically meaningful parameters can suggest issues with the model structure, identifiability problems, or inconsistencies between the model and the underlying mechanisms, even if the model exhibits good statistical fit and predictive power. This mechanistic evaluation provides a crucial layer of validation beyond purely statistical measures, ensuring that the model not only fits the data but also offers a plausible and interpretable representation of the system under study.

Software implementation

We developed our workflow as a unified, Matlab-based pipeline that integrates the previously mentioned three tools for structural identifiability analysis (SIA) alongside estimation and practical identifiability analysis (PIA) methods using AMIGO2 [53]. AMIGO2 (Advanced Model Identification using Global Optimization) is a powerful toolbox for dynamic modeling and optimization, offering a broad selection of nonlinear optimization solvers. These include direct and indirect local methods, multi-start local approaches, global stochastic algorithms, and hybrid optimization techniques. Additionally, AMIGO2 facilitates PIA.

To enhance computational efficiency, particularly for demanding tasks such as parameter estimation, using AMIGO2 in this framework enables automatic generation of C-compiled code, significantly improving performance. Furthermore, we extended the pipeline with additional code to support other key steps, such as stability analysis. Several case studies, based on commonly used models of microbial communities, were implemented and tested within this workflow.

The resulting integrated software is available in both electronic notebook format (live scripts) and as standard MATLAB scripts. It features the following key components:

• Pre-estimation analysis: the workflow simplifies model definition, data integration, and identifiability analysis, ensuring a solid foundation for parameter estimation. This module efficiently handles multi-experiment datasets, incorporates data visualization tools, and facilitates structural identifiability analysis using several methods, so non-expert users can determine whether model parameters can be uniquely inferred.
• Robust parameter estimation: a diverse set of global and local optimization algorithms enhances the reliability and accuracy of parameter estimation. Sensitivity analysis, practical identifiability assessment, and stability analysis help refine parameter estimates. The framework also supports easy cross-validation with additional datasets and leverages parallel computing for improved efficiency and scalability.
• Predictive power analysis: to ensure the model produces reliable and meaningful predictions, the workflow includes sensitivity analysis, statistical goodness-of-fit tests and cross-validation to assess model performance. These methods enable model comparison, interactive plotting, time series visualization, and statistical summarization, aiding in result interpretation. Furthermore, automated report generation and documentation via electronic notebooks streamline reproducibility and facilitate effective communication of findings.

Phase 1: Structural identifiability analysis

We initiated the workflow with structural identifiability analysis (SIA) using GenSSI2, SIAN, and Structural Identifiability. The analysis covered scenarios ranging from ideal (fully observed states, known initial conditions) to realistic (partially observed states, unknown initial conditions). As summarized in Table 2, all models were found to be structurally identifiable under full observation. However, limitations arose under partial observation; for instance, in the GLV2 case, measuring only total biomass or a single species abundance resulted in non-identifiability, consistent with [26] and highlighting how experimental constraints impact parameter determination.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of SIA results.

FO: Fully Observed, PO: Partially Observed, GI: Globally Identifiable, NGI: Non-Globally Identifiable (i.e. only a subset of parameters are identifiable). Last column indicates which methods were successful. Key: 1 = GenSSI2, 2 = SIAN, 3 = Structural Identifiability. Full details, including the partially observed schemes considered, can be found in S1 Text (Sections 7.5.2, 7.6.2, 7.7.2, 7.8.2, 7.9.2 and 7.10.2).

https://doi.org/10.1371/journal.pcbi.1013204.t002

In generalized Lotka–Volterra (gLV) models, there are scenarios in which certain parameters are locally but not globally identifiable. This means that while the parameters can be uniquely determined in a neighborhood of the true values (i.e., locally), there may exist multiple distinct parameter sets that produce exactly the same model output, making it impossible to distinguish between them based solely on the data. In contrast, global identifiability implies that there is a unique parameter set (within the parameter space) consistent with the observed outputs. Understanding and resolving these ambiguities is crucial for reliable parameter estimation and model interpretation.

We note that the presence of parameters that are locally but not globally identifiable often reflects underlying symmetries in the model structure. In particular, permutation symmetries can give rise to multiple distinct parameter sets that produce indistinguishable outputs. However, in our analysis of generalized Lotka–Volterra (gLV) models, we found that when the system is fully observed (i.e., when all species are individually measured over time) the model parameters are globally identifiable under generic conditions. This suggests that the gLV model, in its standard form, does not inherently suffer from non-identifiability due to structural symmetries.

In contrast, the instances in which we observed only local identifiability (or complete non-identifiability) corresponded to partially observed scenarios, such as when only the combined abundance of two or more species is measured. In such cases, the permutation-like symmetries arise not from the model itself, but from the structure of the input–output mapping induced by the observational process. The resulting lack of global identifiability is therefore a consequence of limited or aggregate measurements.

To address these issues, we highlight two common strategies for restoring identifiability: (i) refining the experimental design to provide more informative or disaggregated measurements (e.g., distinguishing between individual species), and (ii) reformulating the model through reparameterization or by introducing biologically or physically motivated constraints that remove unidentifiable degrees of freedom.

The performance of SIA tools varied. GenSSI2’s symbolic approach was effective for simpler models but struggled with larger partially observed ones due to computational demands. SIAN and Structural Identifiability, combining symbolic and numerical techniques, handled more complex models efficiently. The EPHP model could not be analyzed due to its discontinuous formulation, a limitation of current SIA tools assuming continuous dynamics.

Phase 2: Parameter estimation and practical identifiability

This phase addresses the challenges of finding reliable parameter estimates from noisy data, considering potential non-convexity and numerical issues.

Optimization strategies and performance.

We first explored multistart local optimization using least-squares optimizers (lsqnonlin, nl2sol) [53]. This revealed significant challenges across most models, particularly non-convexity (multiple local optima) and model instability (numerical blow-ups). For example, in the GLV3 case with 10% noise (Fig 2), only 17.5% of runs converged, and only 6% reached objective function values near the nominal (true) value. Many runs terminated due to blow-ups or converged to poor local optima (LO) or potentially overfitting solutions (OF).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Case study GLV3: multistart optimization using the nl2sol local solver.

Top figure shows a histogram of the 35 optima obtained in 200 runs, highlighting the presence of overfitting (OF), underfitting (local optimum, LO), and good fit (GF) solutions. The x-axis shows the of the ratio between the objective function achieved and that of the nominal vector of parameters. Figures below present the nominal system behavior and show examples for the OF, GF, and LO cases, respectively.

https://doi.org/10.1371/journal.pcbi.1013204.g002

Overall, multistart local methods showed acceptable performance only for the simplest cases (GLV2) and moderate effectiveness for PCM, achieving 70% good fits. For more complex cases, such as GLV3 and particularly MGLV models, convergence rates were significantly lower. Blow-ups occurred frequently (over 80%, even with tight bounds), and there was a high risk of converging to suboptimal solutions or overfitting. The high-dimensional MGLV model was especially challenging (good fits in only 12% or runs), often failing to accurately reproduce system dynamics, even when using noise-free data. In the CompV and EPHP cases, while most of the multistarts converged, good fits were scarce: only 13% of runs for EPHP and 30% for PCM achieved satisfactory results.

In contrast, the enhanced Scatter Search (eSS) global optimization algorithm exhibited significantly greater robustness. Convergence rates were consistently high (exceeding 95% overall and 85% for MGLV), effectively navigating complex optimization landscapes. While eSS did not completely eliminate challenges such as potential overfitting or local optima traps in highly complex models (e.g., MGLV, GLV3), it consistently delivered much more reliable results than multistart local methods. Additionally, the computational cost using eSS was very reasonable, ranging between 1 and 2 minutes for all cases on a standard PC with Intel i5-13500 CPU (further details available in S1 Text, Section 7.1).

Phase 3: Predictive power analysis

The final phase assessed model generalizability and overfitting risk using cross-validation with held-out data, also incorporating datasets from novel experimental conditions. Our analysis confirmed that overfitting is a significant risk across all models. Solutions achieving excellent fits to the training data often failed to predict system behavior accurately under new conditions. The GLV3 case study provides a clear example (Fig 3): an overfitted solution (OF) matches the training data well but exhibits unstable dynamics long-term (left panel) and performs poorly in cross-validation with different initial conditions (center panel). In contrast, a better-calibrated, though not perfectly identifiable, solution (GF) captures the trends under new conditions more reliably (right panel).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Case study GLV3.

Left figure: An overfitted solution OF that agrees very well with the data but shows oscillatory behavior (and eventually blow-up) after t = 2.0. Center figure: cross-validation of the same OF for different initial conditions, showing very poor predictive value. Right figure: cross-validation of a good fit (GF), showing that due to practical identifiability issues, the agreement with the data is not very good, although ultimately predicts well steady state values at final time.

https://doi.org/10.1371/journal.pcbi.1013204.g003

This underscores the insufficiency of relying solely on goodness-of-fit to training data and highlights the necessity of cross-validation for evaluating true predictive power. Predictive capability generally decreased with increasing model complexity and noise. While simpler models could maintain some predictive power if carefully calibrated, complex models like MGLV showed severely compromised prediction, even when calibrated on noise-free data, linking back to the practical identifiability limitations discussed in Phase 2.

Recognizing that obtaining additional experimental data for cross-validation can be challenging, we also explored several techniques for residual analysis and concluded that Quantile-Quantile (Q-Q) plots can be used as a potential indicator of overfitting based solely on training data (details in S1 Text, Section 7.6.4).

Overall, this phase emphasizes the critical need for rigorous validation beyond the initial calibration dataset to ensure models are not merely fitting noise but capture the underlying system dynamics robustly, especially for complex biological systems like microbial communities.

Overall assessment of the workflow

The initial phase of our workflow uses structural identifiability analysis (SIA) to confirm whether model parameters can be uniquely determined from a given experimental design. The analysis often reveals that while models might be identifiable with perfect data, they become unidentifiable with realistic, limited measurements, such as partially observed scenarios. Without performing SIA, a modeler risks wasting enormous time and computational resources trying to fit a model whose parameters are fundamentally unknowable from the combination of inputs and outputs in the experimental design. They would be attempting an impossible task, leading to non-unique solutions where countless different parameter sets produce the exact same model output. This makes it impossible to draw reliable conclusions, turning the modeling effort into a futile exercise and completely undermining any claim of mechanistic insight. This workflow helps avoid such a dead-end by identifying structural limitations before the intensive estimation process begins.

Following this, the parameter estimation phase of our workflow addresses the dual challenges of finding the best-fit parameters and ensuring those parameters are themselves reliable. The results show that standard, local optimization methods (widely used) often fail for complex models, whereas a global optimizer is far more robust. Critically, even a model that perfectly fits the data can suffer from poor practical identifiability, where parameters have massive uncertainty. By using inadequate optimization tools, a modeler might wrongly conclude that their model is flawed, when in fact the tool was simply not powerful enough to find the global solution. More dangerously, by stopping at a good fit without assessing practical identifiability, a modeler can fall into a major trap, leading to false scientific claims. The model may look perfect, but the estimated parameters could be meaningless, leading to incorrect biological interpretations. For example, reporting a competitive interaction that is actually symbiotic. This mischaracterization of the system’s dynamics can misdirect future experiments and lead to a fundamental misunderstanding of the biological system. Our workflow helps to prevents such erroneous conclusions by demanding that the parameter estimates themselves be trustworthy.

The final phase uses cross-validation to assess a model’s ability to predict outcomes under new conditions, which is the ultimate test of its validity. The analysis confirms that overfitting (capturing noise rather than the true underlying dynamics) is a severe risk, especially in gLV models. Neglecting rigorous validation creates a model that may perfectly describe the training data but is useless for its primary purpose: prediction. When faced with new scenarios, the overfitted model will fail spectacularly, producing inaccurate and unreliable forecasts. This renders the model untrustworthy for any practical application, such as designing a stable microbial consortium or predicting the effect of a treatment. Our workflow helps to avoid this critical failure by ensuring the model is not just a descriptive snapshot of old data, but a robust tool capable of generating reliable predictions about the future.

In this context, it is important to note that several studies have highlighted fundamental limitations of generalized Lotka-Volterra (gLV) models in microbial community dynamics. Momeni et al. [58] critique the universality assumption of pairwise Lotka-Volterra models, showing through mechanistic references that diverse chemical-mediated interactions demand tailored mathematical forms, as mismatched models yield erroneous predictions on dominance or coexistence. Similarly, Hart et al. [59] found that parameters from standard batch cultures failed to predict growth in a synthetic cooperative yeast community due to environment-dependent phenotypes like metabolite rates. Picot et al. [14] further argue that gLV’s assumption of dynamic equilibrium clashes with batch cultures’ static resource-exhausted endpoints, fostering overfitting to early growth phases and unreliable long-term predictions with setup-sensitive parameters.

While these studies point to fundamental problems with Lotka-Volterra models, our work identifies additional, critical challenges. We reveal that these models might also suffer from structural and practical non-identifiability, local optima (underfitting), and numerical instabilities. Our research therefore complements previous findings and serves as a clear warning: the popular gLV framework has numerous pitfalls, and its application to dynamic microbial modeling requires significant caution.