Mann-Whitney U test results on synthetic data

Features extracted from sequences with each label exhibit non-normal distribution (see Figs B-E for 22SF and Figs F-I for 5EWSI in S1 Text). Therefore, we conduct the Mann-Whitney U test, which is non-parametric and does not assume specific data distribution, on 22SF and 5EWSI computed from “T” and “N” replicates. We find statistically significant differences between features derived from “T” sequences and those from “N” sequences with p ≪ 0 . 001. These findings indicate the potential to statistically discriminate between these two categories based on 22SF or 5EWSI.

Performance on withheld testing set

We evaluate the classification performance of synthetic-data-trained classifiers on withheld testing sets, as presented in Fig 3 (detailed data are provided in Table D in S1 Text). Results show that all the classifiers can achieve near-perfect performance on withheld testing sets, with AUC scores ranging from E5K (0 . 9911 ± 0 . 0038) to D22G, D22K, D22L, D22S, D5G, D5K, D5L, D5S, W22L, W22S, W5G, and W5L (1 ± 0 . 0000). Given the high AUC scores, we generated t-distributed stochastic neighbour embedding (t-SNE) plots (Fig J in S1 Text), which visually confirm the clear distinction between the two classes, supporting the near-perfect classification performance on the withheld testing set.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The AUC values of 32 synthetic-data-trained classifiers (horizontal axis, see Fig 2) on withheld testing sets.

Classifiers are reordered by AUC scores. Error bars correspond to the 95% confidence intervals. DeLong tests are conducted to compare the AUC values of classifiers, with detailed results available in Tables A-D in S2 Text (where the predictive model is fixed) and Tables E-L in S2 Text (where the training data is fixed).

https://doi.org/10.1371/journal.pcbi.1012782.g003

The AUC scores of 32 different classifiers are consistently high, exceeding 0 . 9900 and being numerically close. Hence we conduct DeLong tests [34] to compare the AUC scores of different classifiers. Tables A-D in S2 Text present p-values of the DeLong tests across eight training sets with a predetermined model (GBM, LRM, KNN, and SVM, respectively). Classifiers trained using 5EWSI of (i.e., E5G, E5L, E5K, or E5S) generally exhibit slightly lower AUC scores compared to classifiers trained using the same predictive model but with different datasets (p < 0 . 001 for most classifiers).

Tables E-L in S2 Text present the DeLong test results among four models, trained on the same dataset. Counterintuitively, despite the distinct properties of various predictive models and their ability to handle different types of data, the test results suggest no significant difference in AUC scores among four predictive models when applied to most of the synthetic testing sets. However, there are some exceptions: when using 22SF of (see Table F in S2 Text), the AUC of four classifiers, E22K has the lowest AUC score  ( 0 . 9956 ± 0 . 0027 )  compared to E22S (0 . 9993 ± 0 . 0011, p < 0 . 01), E22L (0 . 9993 ± 0 . 0011, p < 0 . 01), or E22G (0 . 9991 ± 0 . 0012, p < 0 . 01). Moreover, as illustrated in Fig 3, classifiers trained by 5EWSI of across four models rank the last four in terms of AUC scores, with E5K (0 . 9911 ± 0 . 0038, p < 0 . 05) being the lowest, while the AUC scores among E5G, E5S, and E5L are statistically similar (p > 0 . 05). This means 5EWSI of is the most challenging one to be distinguished among the four models.

Performance on withheld testing set with varying sequence length and gap

In this context, replicates end at the transition points, leaving no response time for implementing mitigation measures. Despite the satisfactory performance of 32 classifiers over withheld testing sets, two questions naturally emerge: (1) How early can we predict an impending disease outbreak? (2) How much data is required for prediction to maintain a certain level of accuracy? To address these concerns, we conduct two experiments using the Rolling window (where the distance between data and the transition point varies while fixing the data length) and the Expanding window (where the data length varies while the distance between data and the transition point is fixed), as illustrated in Figs 4A and 5A. We repeatedly conduct training-testing processes using Rolling windows or Expanding windows, resulting in corresponding classifiers at each round. The AUC values and classification accuracy are depicted by points using different colors and shapes. Results are shown in Figs 4 and 5 for AUC and Fig L in S1 Text for accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Illustration in Rolling window experiment settings and AUC score at different reiterations.

A: Rolling window settings on and . We maintain a fixed length of input time series L and roll the window L from the right to the left boundary. B-I: AUC scores depicting the classification performance of the classifier trained from changing gaps D between the subsequence and transition points. Two feature extraction methods (22SF, 22 statistical features; 5EWSI, 5 early warning signal indicators) are put in two rows and four predictive models (GBM, gradient boosting machine; LRM, logistic regression model; KNN, k-nearest neighbor; SVM, support vector machine) are presented in four columns.

https://doi.org/10.1371/journal.pcbi.1012782.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Illustration in Expanding window experiment settings and AUC score at different reiterations.

A: Expanding window settings on and . We fix the length of D as 30, then expand L until reaching the left boundary of sliced data. The initial length of L is 5. B-I: AUC scores depicting the classification performance of the classifier trained from changing gaps D between the subsequence and transition points. Two feature extraction methods (22SF, 22 statistical features; 5EWSI, 5 early warning signal indicators) are put in two rows and four predictive models (GBM, gradient boosting machine; LRM, logistic regression model; KNN, k-nearest neighbor; SVM, support vector machine) are presented in four columns.

https://doi.org/10.1371/journal.pcbi.1012782.g005

The results of the Rolling window (Figs 4B–4I) illustrate that approaching the transition point can generally increase the AUC scores for all classifiers, among which those trained from the GBM model showing better performance across four types of noise data. Classifiers trained on 5EWSI of any noise data can achieve AUC scores of almost 1 when the gap is shorter than 50, while those employing 22SF require a distance of less than 30 to reach similar performance as 5EWSI. Classifiers trained on 22SF of and slightly outperform those trained on the other two noise data for all four models (see blue triangle and purple round points in Figs 4B–4E. However, 5EWSI of and exhibit notably lower AUC across predictive models when the distance is greater than 100 (see blue triangle and green square points in Figs 4F–4I. We also observe that when the distance between the time series to the transition point is less than 100, two feature extractions computed from any of the sliced time series exhibit similarly excellent performance across four models. As the window moves farther from the transition point, at a distance of 300 data points for example, the GBM model can achieve better performance for both and in terms of feature extraction. We further notice that classifiers trained from 5EWSI of or can achieve better and more robust performance as their AUC values do not increase substantially along with closer distances (see yellow cross and purple round points in Figs 4F–4I. In contrast, classifiers trained from 5EWSI of and yield significantly improved classification performance when far from the transition points (see green square and blue triangle points in Figs 4F–4I). Additionally, it is worth noting that the results of LRM and SVM models present negligible differences (less than 0 . 001 in general).

The results of the Expanding window experiments are presented in Figs 5B–5I. AUC scores exhibit a rapid improvement as we increase the length of the testing time series from 5 to 100, preserving an AUC score of over 0 . 97 thereafter, which suggests that the classifiers derived from the framework can remain applicable even for short synthetic time series. While longer time series data intuitively contribute to better performance, we observe a slight decreasing trend in AUC for classifiers trained by 5EWSI when the input time series length approaches 370, likely due to redundant data distant from the transition point. Interestingly, classifiers trained from 22SF of the original time series show no such decline in AUC. One speculation is that classifiers trained on data represented by 22SF are more robust when dealing with data redundancy.

Performance on empirical testing set

Although our models perform well on withheld test sets, we need to further evaluate their practical utility using real-world datasets. Performance on empirical datasets is also the criterion for model selection as it provides insight into the optimal choices for the model of simulation, feature extraction, and the machine learning algorithm within the framework.

We tested the performance of 32 simulation-trained classifiers (see Fig 2) on four empirical datasets: two at the national level, COVID-19 data from Singapore () and from 18 countries (), and two at the city level, COVID-19 data from Edmonton, Canada (), and SARS data from Hong Kong (). Although violating the assumption of homogeneous population distribution in the SIR model, we utilize to evaluate the performance of classifiers on data that deviates from the model assumption. We use the to compare the performance with existing methods. Note that three datasets (, , and ) solely contain replicates labeled as either “T” or “N”, making it impossible to compute AUC scores. As a result, we use the classification accuracy as the performance measurement on testing sets, and the results are presented in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Classification accuracy of 32 synthetic-data-trained classifiers (horizontal axis, see Fig 2) to correctly classify subsequences of four empirical datasets.

Error bars correspond to the 95% confidence intervals.

https://doi.org/10.1371/journal.pcbi.1012782.g006

As shown in Fig 6, the performance varies across the four empirical datasets depending on the type of data used for model training, specifically the simulations generated from the SIR model with different types of noise. Notably, there are two models, E5L and W 5L, that consistently demonstrate high classification accuracy across all empirical datasets, making them the best-performing models among the 32 candidates. These findings suggest that simulating the SIR model with either white noise or environmental noise is preferable. Additionally, computing five classic early warning indicators from the data and using the logistic regression model would provide better classification performance.

For dataset , 17 out of 32 models performed better than random with 16 achieving an accuracy of 0 . 9 or higher. For dataset , 18 models achieved an accuracy above 0 . 5, but only 9 models exceeded 0 . 9. Although and are both national-level datasets, we observed generally lower accuracy on compared to , which is consistent with our rationale that Singapore data is more suitable for SIR model assumptions. The performance of , and , or in other words, between replicates solely labeled as “T” (outbreaks) or “N” (non-outbreaks) seem to be complementary. Most classifiers performing well in classifying “T” replicates fail to predict replicates labeled as “N”, except for two classifiers, E5L and W5L, achieving an accuracy of 1  ( ± 0 . 0000 )  in out-of-sample empirical data classification tasks. For dataset , which is also used in Chakraborty et al. [32] as the empirical testing set, the best two models (AUC = 1) outperformed the models trained by classic early warning signals (AUC  <  0.5), the deep learning model proposed by Bury et al. [26] (AUC  <  0.5), and Chakraborty et al. [32] model (AUC = 0.71). Note that the error bar for is significantly larger compared to the other three datasets as its sample size is much smaller.

Although the minimum data length of and is 14, we further access the performance of the classifiers on shorter replicates by removing 7 data points (i.e., ending the data one week before the outbreak timing) from these two COVID-19 datasets. Results presented in Tables M and N in S1 Text show that if the disease outbreak occurs seven days later, there is no significant difference between using the current data for prediction and additionally incorporating data from the next seven days. Thus, we can effectively predict the outbreak with a lead time of one week. It is worth noting that the “N” replicates are selected from the subset of SARS data with , with random lengths, hence we do not conduct the same test on shorter replicates for this dataset. We should also acknowledge that asymptomatic or presymptomatic individuals exhibit scant influence on flu transmission [35], in contrast, COVID-19 incidence data are significantly underestimated due to these “unnoticeable” infections. We expand the empirical data on individuals infected with COVID-19 five-fold and repeat the experiment. The results in Tables O and P in S1 Text show that underestimation of incidence data does not have a significant impact on classification performance, no matter whether the original performance is excellent or undesirable.

Time series classification and predictive models

For a univariate time series I(t), feature-based methods perform a feature extraction from I(t) before the classification procedure. The computed features serve to transform the time series into a structured format comprising N numeric variables, allowing us to directly employ machine learning algorithms to classify the feature data, and subsequently accomplish TSC tasks. Here, we employ four predictive models, namely GBM, LRM, KNN, and SVM, for classification use. Among these, GBM and KNN are non-linear classifier models, LRM is a linear model, and a non-linear kernel was chosen for SVM. We additionally considered the simple decision tree (SDT) as it is more interpretable than GBM. However, SDT-trained models perform less effectively on withheld data than GBM-trained models, we therefore focus on the GBM. Detailed results of SDT, as well as the importance of features, are provided in Table E and Fig K in S1 Text. AUC scores [44] of classifiers and classification accuracy on testing sets are recorded as performance measures.

In this work, the code and analysis are carried out using the R. Two models, GBM and KNN, are implemented using the caret package [45]. The SVM model is applied using the e1071 package [46]. The LRM model, which is the logistic regression model in our work, is computed by the LRM function contained in the stats package of R [47]. The AUC of each classifier is calculated using the pROC package [48].

Synthetic data used for training and testing

We first elaborate on how the synthetic data are simulated from dynamic models. Then we explain how the simulated time series are pre-processed for classification use. Note that synthetic data are used for both the training and testing processes in this paper.

Synthetic time series generation.

We start the investigation on the classic susceptible-infectious-recovered (SIR) model [31]:

(1)

where S, I, and R are susceptible, infected, and recovered individuals at time t, respectively. Λ is the recruitment rate of susceptible population, β ( t )  is the transmission rate at time t, μ is the death rate, and α is the recovery rate.

Admittedly, the SIR model is often regarded as simplistic in capturing the dynamics of infectious diseases. More complex and context-specific models are better suited for simulating the intricate transmission mechanisms in real-world scenarios. But it is also the simplicity of the SIR model that provides the “greatest common factor” to accommodate the novel disease to some extent. Furthermore, the dynamics of infectious diseases can be easily influenced by random internal and/or external disturbances. To augment the SIR model, we followed Chakraborty et al. [32] and introduced three distinct stochastic differential equations (SDEs) incorporating white noise, multiplicative environmental noise, and demographic noise to generate synthetic incidence time series. The rationale behind selecting these specific noise types, as well as the parametrization, are justified in Chakraborty et al.[32].

Mathematically, the basic reproductive number , a dimensionless value representing the expected count of secondary infections caused by a single infectious individual in a completely susceptible population, serves as an important metric for the stability of equilibrium points in the SIR model [49]. When , the disease-free equilibrium is stable, and the system remains in a non-outbreak state. indicates the potential for the disease to persist within the population, and the endemic equilibrium becomes stable. At the critical threshold of , a transcritical bifurcation occurs, wherein the disease-free equilibrium and endemic equilibrium intersect and exchange their stability properties. Following the methodology proposed by Chakraborty et al. [32], all parameters except the transmission rate β are to be constants. Then the value of becomes solely dependent on β ( t ) , expressed as , with . We further assume a linear variation in the transmission rate β with respect to time t: , where and follow triangular distributions. Starting with a random initial value of β ( t = 0 )  such that , we gradually increase β over time to make exceed 1 to obtain the outbreak simulation. During this process, the presence of noise ensures that the infected individual I increases after the critical point. Without noise, I would become and remain zero when is initially less than 1. If the slope is sufficiently small, not exceeding the critical value of reproduction number within the time interval t ∈ [ 1 , 1500 ] , the simulation results in null bifurcation.

Empirical data used for testing

To assess the applicability of the framework in real scenarios, we apply empirical time series data of two infectious diseases: the novel coronavirus disease (COVID-19) and severe acute respiratory syndrome (SARS). COVID-19 has emerged as an excellent subject for novel disease studies due to the abundance of accessible data, both temporally and spatially, and its prolonged duration. COVID-19 exemplifies novel infectious diseases causing significant outbreaks (transcritical bifurcation). The SARS outbreak was initially identified in November 2002 [50]. Since 2004, there have been no known cases of SARS reported worldwide, providing an illustrative example of non-outbreak (null bifurcation) in our study.

Similar to synthetic data, we apply the effective reproduction number () as the criterion to determine whether empirical time series exhibiting outbreaks (transcritical bifurcation) and the timing of outbreak occurrence. Given a full-time series I exhibiting outbreak(s), is labeled as “T” if and only if for any , , and . For I exhibiting the non-outbreak, any subsets will be labeled as “N”.

(# replicates = 19; COVID-19 daily infection data from Singapore) The fundamental assumption of the classic SIR model is the homogeneity of population distribution (i.e. each individual in the population has an equal contact probability). However, this assumption may not hold for data spanning a geographically extensive area, such as national-level data, despite their potential for greater accuracy and availability. Nonetheless, the COVID-19 data from Singapore presents an ideal subject for study, where the population demographics align well with the assumptions of the SIR model. Additionally, the government responded promptly to the pandemic, implementing intensive tracking via mobile technology (Government Technology Agency of Singapore [51]), enhancing data accuracy. To maintain the consistency of the SIR model and the empirical testing set, we focus on the COVID-19 infection data from Singapore, spanning from January 26, 2020, to February 18, 2024. The data is collected weekly and then smoothed by evenly distributing the infection number over seven days of the corresponding week to obtain daily data. is estimated via R package EpiEstim [52] with a mean value of 6 . 3 and a standard deviation of 4 . 2 for the serial interval [53]. Replicates with lengths shorter than 14 days are excluded, resulting in a testing set comprising 19 replicates labeled as “T”.

(# replicates = 14; COVID-19 daily infection data from Edmonton, Canada) The dataset was obtained from the City of Edmonton’s Open Data Portal [54] and includes daily incidence data from March 2020 to July 2023. Similar to , the was estimated using R package EpiEstim [52], and subsequences with and length is larger than 56 (8 weeks) were selected. Note that this empirical dataset and the data processing are identical to the empirical datasets used in [32] for comparison purposes.

(# replicates = 194; COVID-19 incidence data from 18 countries) While the national data are often underestimated (due to presymptomatic and asymptomatic case) and against the homogeneity of the population, we still use the national level data to test the robustness of the model. To derive the replicates and the corresponding labels, we apply the estimation from Huisman et al. [55] for COVID-19 incidence data, where the assumption is that the incidence data come from symptomatic individuals. Given the typical lag between disease occurrence and data collection, subsequences with lengths shorter than 14 days (i.e., two weeks), or replicates containing missing data are excluded. The final dataset comprises 194 replicates, labeled as “T”, encompassing data from 18 countries. Detailed information on the sliced data is provided in Table H in S1 Text.

(# replicates = 1200; SARS 2003 incidence data from Hong Kong) We utilize SARS data from Hong Kong, spanning from March 17 to July 11, 2003 [56]. The original data comprises Cumulative Cases, Death Cases, and Recovered Cases. To maintain consistency with the SIR model, we computed the prevalence (i.e., the “I” term in the SIR model) by subtracting “Death Cases” and “Recovered” from Cumulative Cases. The recorded data displays missing values, with one missing data point every Sunday for the initial 11 weeks and two missing data points every Saturday and Sunday in the last 5 weeks (see Fig O in S1 Text). To impute the missing data, we applied R package imputeTS using na_interpolation and obtained the completed incidence time series. The mean and standard deviation of the serial interval are 8.4 and 3.8, respectively [57]. We found that the mean value of estimated falls below 1 during the estimation interval from April 18 to April 24, 2003, and never exceeds 1 afterward. To ensure greater confidence in the values being less than 1 for the subsequences, we designate the sequence after a later date, May 15, 2003, as the null bifurcation scenario and randomly select 1200 subsets, each with a random length larger than 7, as the “N” replicates in the testing set.

Statistical features and early warning signal indicators

Classifying raw time series, especially those observed at higher frequencies, usually involves working with high-dimensional data, which can be computationally intensive and less interpretable [58]. To address this concern, people seek a proper representation of the original time series, typically encompassing statistical attributes, in a numerical format. In this paper, we employ two distinct sets of features, 22SF and 5EWSI, to represent the sequence for classification purposes.

Numerous alternative features for time series are available, and among them, 22SF computed through R package Rcatch22 stands out as a promising choice, as it exhibits ideal performance in TSC and minimal redundancy [59]. The 22SF utilized in this study are detailed in Table B in S1 Text, and more information is available in Lubba et al. [33]. Note that the 22SF of the simulated time series do not follow normal distribution, therefore, we conducted the Mann-Whitney U test on these features computed from “T” and “N” sequences across four types of noise data. The results are depicted in Figs B-E in S1 Text. As can be easily observed, the features computed from data with two labels are significantly different, with p ≪ 0 . 001. These findings suggest that despite the visually indistinguishable nature of and , these two types of sequences can be theoretically separated by examining these statistical features.

Alternatively, we consider 5 early warning signal indicators (5EWSI) from time series to serve as the features, with their formulas provided in Table C in S1 Text. Notably, there is no overlapping between 22SF and 5EWSI. We also implemented the Mann-Whitney U test, and results demonstrate that there is a difference in these five features computed from time series labeled as “T” and “N” (Figs F-I in S1 Text).

Experiment settings for earliness and accuracy

Maximizing classifier accuracy using the complete time series is typically the primary goal of TSC problems [60]. However, in time-sensitive applications, such as infectious disease monitoring and pandemic forecasting, the aim shifts to optimizing the balance between two contradictory (or contradicting or opposing) objectives: accuracy and earliness in the classification task. Such transition implies that the classifier should ideally handle inadequate (shorter time intervals) and early (those far from the impending transition) sequences while maintaining a certain level of classification accuracy to provide reliable references for policymakers. Consequently, critical questions arise in two-fold: How early can predictions be reliably provided? What is the sufficient length of testing time series required to offer a reliable prediction? To address these questions, we conducted two experiments, as illustrated in Figs 4A and 5A.

Firstly, we employ the Rolling window approach on both and . We maintain a fixed length of 100 for input time series L and “roll” the L from right to left, resulting in an increasing gap D between the last time point of the window and the transition point T. Starting with an initial gap of 0, we increment the gap by 5 units for each iteration k, yielding 61 experiments. After determining the input time series L for each round, we follow the same feature-computation and data partition procedure outlined in section Synthetic Time Series Generation and train classifiers through four models. We record the AUC and classification accuracy on testing sets for each classifier at every iteration.

To further investigate predictive performance using varying lengths of the time series data, we conduct training and testing processes using an Expanding window approach. Initiating with a 5-time-point input time series L, we expand the L by adding five more data points to the left end of the time series for each iteration, while maintaining the length of gap D as 30. Consequently, the length of input time series L ranges from 5 to 370, resulting in 74 reiterations for each time series dataset and each model. We again follow the feature extraction, and training-testing sets partition approaches, and perform the classification. We repeat these experiments across all eight datasets and four models, recording the corresponding AUC scores and accuracy values for assessments.