Notations and copula-based distance

Let be the i-th time series (1 ≤ i ≤ n), where T_i is its length and n is the number of time series. We assume that these time series are strictly stationary drawn from J₀ clusters, and the time series in each cluster share a common dynamic pattern. The purpose is to identify these J₀ clusters. We propose a method based on the copula function to represent the dynamic pattern of the time series. Specifically, for a fixed positive integer h, (X_ij, X_i(j+h)) and (X_i′j, X_i′(j+h)) have the same copula function if time series X_i and X_i′ belong to a common cluster.

For random variables X and Y with continuous marginal cumulative distribution functions F_X(x) and F_Y(y), respectively, we denote the joint cumulative distribution functions by F(x, y). Sklar’s theorem [19] claims that a unique copula function C(u, v) exists connecting F(x, y) to F_X(x) and F_Y(y) via F(x, y) = C(F_X(x), F_Y(y)), which is equivalent to . This means that the copula function C(u, v) can capture the structure of dependence between random variables X and Y. We use the copula to capture the dynamic pattern of the time series.

For each 1 ≤ i ≤ n, we denote by C_i,h(u, v) the copula function of (X_ij, X_i(j+h)). For arbitrary i ≠ i′, we define the following copula-based Cramér-von Mises distance to measure the dissimilarity between time series X_i and X_i′: (1)

The copula-based distance D_h(i, i′) satisfies the following three classical properties:

1. (Nonnegativity) D_h(i, i′) ≥ 0. Moreover, D_h(i, i′) = 0 if and only if (X_ij, X_i(j+h)) and (X_i′j, X_i′(j+h)) share a common copula function.
2. (Symmetry) D_h(i, i′) = D_h(i′, i).
3. (Triangle inequality property) D_h(i, i′) ≤ D_h(i, k) + D_h(k, i′).

Nonnegativity and symmetry are apparent in the foregoing, and the triangle inequality property can be obtained from the Cauchy–Schwarz inequality. Traditionally, the dependence structure of time series is often captured using an autocovariance-based linear correlation, which fails in nonlinear dependency structures. Nonlinear dependence may be caused by various nonlinear structures. The copula function does not make any assumptions about the model, and is a flexible method that can capture them. Moreover, copula-based distance is not affected by strict monotonic transformations (i.e., logarithmic transform and exponential transform) due to the property of mapping invariance of the copula function [18].

If time series X_i(1 ≤ i ≤ n) are dependent in the order of one, we let h = 1 and directly use D₁(i, i′) as the dissimilarity measure of X_i and X_i′. If the times series are dependent in a higher order, the dissimilarity measure of X_i, X_i′ can be defined as the following weighted version: where K is the highest order of dependence of time series X_i and X_i′ considered in the dissimilarity measure definition, and ω_h is the weight of D_h(i, i′). We can allow ω_h to decrease as h becomes larger.

The selection of K depends on the unknown underlying model. We may entertain several possible values of K and use model selection criteria such as AIC and BIC to select the optimal value of K. On this point, [20] claimed that the aim is not the goodness-of-fit to the underlying models but clustering them properly. We thus do not make any significant effort on this issue. In practice, the value of K should be chosen with specific knowledge of the application. It is often the case that the strongest serial dependency occurs in small lags, and a larger value of K means greater computational time and redundant information. Moreover, we can always obtain reasonably satisfactory results using small values of K in our simulations and applications. A large number of studies focusing on application have shown that it suffices for a large number of time series to use a lag of one to obtain goodness of fit. Therefore, in practice K = 1 is highly recommended.

Estimation of copula-based distance

In practice, the copula function is usually unknown, and thus the copula-based distance (1) cannot be used directly. We propose a nonparametric estimation of the copula function that can be plugged into (1) to obtain the estimated distance.

Specifically, we denote the empirical distribution functions of (X_it, t = 1, ⋯, T_i) by , where I(⋅) is an indicator function. For 1 ≤ t ≤ (T_i − h), we further define Then, the nonparametric estimator for the copula function of (X_it, X_i(t+h)) (i.e., C_{i, h}(u, v)) is defined as We replace C_{i, h}(u, v) in (1) by and obtain the corresponding copula-based distance estimator: (2)

If we further assume that the time series are α mixing processes, are strong consistency estimators for C_i,h(u, v) and D_h(i, i′), respectively. We summarize the results as the following theorem:

Theorem 1 Assume that time series X_i = (X_it, 1 ≤ t ≤ T_i)(1 ≤ i ≤ n) are strictly stationary α mixing processes. Then, as T_i → ∞ and , as T_i, T_i′ → ∞, where denotes convergence with probability 1.

The proof of Theorem 1 is provided in Appendix A. In practice, we need to calculate the value of , which is easy. For the details of the calculation of , see the following proposition:

Proposition 1 For i, i′ = 1, ⋯, n, define

Then, for the copula-based distance estimator , (3)

The proof of Proposition 1 is provided in Appendix B.

Clustering algorithms based on copula distance

Once the pairwise distance between the time series has been obtained, we can apply any clustering method that uses a distance matrix as input, such as partition around medoids (PAM) [21], spectral clustering [22], and hierarchical clustering [23]. In PAM methods, we need to find a centroid time series in each cluster to represent the group. PAM methods require that the user specify the number of clusters. Through a hierarchical clustering algorithm, we can obtain a tree dendrogram built starting from the leaves and combining clusters to the trunk. The hierarchical clustering algorithm applied in this paper can be summarized as follows:

Clustering Algorithm

1. Step 1. For i, i′ = 1, ⋯, n and i ≠ i′, compute the estimator for the copula-based distance between time series pair X_i and X_i′. Then, we can obtain the distance matrix. Treat each time series as its own cluster.
2. Step 2. For J = n, n − 1, ⋯, 2:
   1. a. Compare all pairwise dissimilarity measures among the J clusters and identify the pair of clusters most similar. Merge them into as one.
   2. b. Compute the pairwise dissimilarities of the new J − 1 clusters.

It is worth mentioning that dissimilarities should be defined among clusters based on distances between time series prior to applying the clustering algorithm. Single linkage, complete linkage, average linkage, centroid linkage, and Ward’s linkage are the most common methods to extend dissimilarities among observations to dissimilarities among clusters [24].

In practice, The computations of clustering can be performed in a more efficient way. [25] introduced a recurrence formula which can be used to compute the updated inter-cluster distance values efficiently. If a clustering procedure does not satisfy such a recurrence relation, the initial data should be retained throughout the entire process when updating cluster distances. All these aforementioned linkage methods fit into the Lance-Williams formalism, and can therefore easily be implemented with user-defined time series distance. In this paper, the efficient method of implementing clustering is to store the matrix of copula distances and update inter-cluster distance using Lance-Williams recursive formula.

Specifically, the traditional Ward’s linkage method minimizes the increase in total within-cluster sum of squared error. This increase is proportional to the squared Euclidean distance between cluster centers. Here, we extends the classical Ward’s linkage method that relies on Euclidean distance to copula distance, which still shares the same parameters in the update formula with original Ward’s linkage method [26]. With the Lance-Williams formula, we do not need to assign the cluster center to compute the new inter-cluster distance.

Recall that we assume these time series are drawn from J₀ clusters. We denote these J₀ clusters by . Let C_j0,h(u, v) be the common copula shared by the time series in the j-th cluster. We further define which is the copula-based distance between the j-th and the j′-th clusters. Let ϵ = min_j≠j′ D_0,h(j, j′), which is the minimal distance between the clusters. Then, for the foregoing clustering algorithm, the following theorem holds:

Theorem 2 Assume that time series X_i = (X_it, 1 ≤ t ≤ T_i)(1 ≤ i ≤ n) are strictly stationary α mixing processes and ϵ > 0. For J > J₀ in Step 2 of Clustering Algorithm, the obtained J − 1 clusters are denoted by . Then, as min_i{T_i}→∞, with probability 1 we have that for every with 1 ≤ j ≤ J − 1, there exists a j′ such that .

We call the results of Theorem 2 the consistency of clustering. The proof of Theorem 2 is provided in Appendix C. If J₀ is known, we can directly cluster the time series into J₀ groups. However, the number of clusters J₀ is usually unknown, and we should detect the optimal number of clusters in practice. A useful approach to determine this optimal number is the silhouette method [27]. For each time series X_i with i = 1, 2, …, n, its silhouette width s(i) is defined as where a(i) is the average copula distance between series X_i and all other time series of the cluster to which X_i belongs, and b(i) is the average copula distance between series X_i and time series in neighboring cluster, i.e., the nearest cluster to which it does not belong. Let the set of time series be partitioned into J clusters. The corresponding average silhouette width is defined as

The average silhouette of the clusters is calculated according to the number of clusters. A high average silhouette width indicates good clustering. Thus, the optimal number of clusters J is that which maximizes the average silhouette over a range of possible values for J. We choose the number of clusters as J*, which yields the maximum value of Sil(J).

Example 1: Nonlinear time series clustering

In this example, we considered the following four models:

1. Threshold autoregressive (TAR) model
2. Exponential autoregressive (EXPAR) model
3. Linear moving average (MA) model
4. Nonlinear moving average (NLMA) model

where the error process ε_t independently followed N(0, 1). These models were used in [30] for linearity tests and [17] to study time series clustering. Except for model (3), the others were conditional mean nonlinear models. The stationarity of these models can be guaranteed. We here only take the TAR model as an instance to illustrate its stationarity. For a TAR model where {ε_t} are independent and identically distributed with zero mean and a finite variance. Then the necessary and sufficient condition for the strictly stationarity to above TAR model when γ = δ = 0 is α < 1, β < 1 and αβ < 1 [31–34]. In model (1), we set γ = δ = r = 0, and α = 0.5, β = −2, hence the necessary and sufficient condition holds, and furhter the stationarity of the TAR model (1) is guaranteed.

We generated four time series from each model, and thus the sample size was 16. We set the lengths of all series as a common parameter T, and we considered two values of T (i.e. T = 100, 200). For sake of simplicity, in all of the experiments we used uniform weight for copula distance, that is w_h = 1 for 1 ≤ h ≤ K. The experiment was repeated 100 times using all the aforementioned distance measures. The clustering similarity indices were calculated and summarized by the boxplot in Fig 1. The distance based on the copula function always yielded the best performance. For each of the distances, a larger series size seems to benefit more. When the series length T = 200, the similarity indices of the copula distance were almost equal to one, which meant that the copula distance could cluster the series into the true group from which they were generated. The first eight distances (d_ACF ∼ d_LNP) needed the assumption of linearity of the time series. Therefore, their results were significantly worse than those of the other nonparametric distances (d_DLS ∼ d_ISD) in this example.

Fig 2 shows the multidimensional scaling (MDS) plot [35] used to visualize observations in two dimensions (T = 200), where the dissimilarity among the time series was based on our proposed copula distance. With the dissimilarity measures obtained from the data, the MDS plot sought a lower-dimensional representation of the data that preserved pairwise distances as closely as possible. Fig 2 shows a clear separation of the four clusters and the capability of copula distance to discriminate among them. Furthermore, it is evident that the series from the MA models and NLMA models are closer to one another because both models expressed time series as a function of white noise.

To gain further insights into copula distance, based on Example 1, we designed two more challenging clustering tasks. The first involved increasing heterogeneity within each cluster and the second explored the performance of copula distance when the strength of nonlinear dependence was changing.

Example 2: Nonlinear time series clustering with increased intra-cluster heterogeneity

In this example, we considered models similar to those in Example 1. The length of each time series was set to T = 200 here. To enhance the difficulty of nonlinear time series clustering, we made some changes to each model. For model (1), TAR model is generalized to the Smooth Transition Autoregressive (STAR) model, which allows for higher degree of flexibility in model parameters. Similarly, for model (3), linear MA model was replaced by the Smooth Transition Moving Average (STMA) model. For models (2) and (4), instead of using a fixed constant in each model, we used varying coefficients generated randomly from some given uniform distributions. Specifically, the models considered were:

1. Smooth transition autoregressive (STAR) model where F(X_t−1) = (1 + exp(−X_t−1))⁻¹ is the smooth transition function.
2. Exponential autoregressive (EXPAR) model
3. Smooth transition moving average (STMA) model where the smooth transition function is specified as
4. Nonlinear moving average (NLMA) model

The results are shown in Fig 3. As the heterogeneity in each group increased, almost all of the distance measured led to different levels of performance degradation except copula distance. The copula distance still generated the best performance among all measures, and its performance has hardly any degradation.

Example 3: Nonlinear time series clustering by adjusting nonlinear strength

In this example, we studied the sensitivity of copula distance to the nonlinear strength of the time series with the length T = 200. We wanted to determine the clustering performance of copula distance when the nonlinear strength of the time series varied. We controlled the strength of nonlinearity by adjusting the coefficients of the models considered in Example 1 as follows:

1. Threshold autoregressive (TAR) model
2. Exponential autoregressive (EXPAR) model
3. Linear moving average (MA) model
4. Nonlinear moving average (NLMA) model

We can see that model (3) remains identical to that in Example 1 and, if (b₁, b₂, b₄) was equal to (2, 10, 0.8), the remaining three models were identical to those in Example 1. On the contrary, if we set (b₁, b₂, b₄) equal to (−0.5, 0, 0), the models degenerated to the following linear models:

1. X_t = 0.5X_t−1 + ε_t;
2. X_t = 0.3X_t−1 + ε_t;
3. X_t = ε_t − 0.4ε_t−1;
4. X_t = ε_t − 0.5ε_t−1.

We can see that the strength of nonlinear dependency decreased when (b₁, b₂, b₄) changed from (2, 10, 0.8) to (−0.5, 0, 0). Therefore, if we assigned (b₁, b₂, b₄) = (2.5α − 0.5, 10α, 0.8α), 0 ≤ α ≤ 1, the size of α represented the strength of nonlinear dependence. The larger the value of α, the stronger the nonlinear dependence. When α = 1, this strength was the highest, which is the situation in Example 1. On the contrary, when α = 0 the strength of nonlinear dependence was the weakest, which was linear. We provided six uniformly spaced values in [0, 1] to α (i.e., α = 0, 0.2, 0.4, 0.6, 0.8, 1), and the results of clustering are shown in Fig 4 below.

From Fig 4, we see that when the time series were simulated from linear models, i.e., α = 0, the distance measures based on the assumption of model linearity yielded the best performance, such as d_PACF, d_PIC, and d_M. The performance of copula distance was not inferior to that of the other methods. As nonlinear strength increased (i.e., α increased), the advantage of nonparametric distance measures become ever more apparent while the clustering performance of methods based on linearity degenerated. In most cases, copula distance yielded far better results than the competition.

Case 1: Annual real GDP data analysis

We considered data concerning the gross domestic product (GDP) obtained from https://www.conference-board.org/retrievefile.cfm?filename=Output-Labor-and-Labor-Productivity-1950-20111.xls&type=subsite. It contained the annual real GDP of the 23 most developed countries in the world from 1950 to 2011: Austria, Belgium, Denmark, Finland, France, Germany, Greece, Iceland, Ireland, Italy, Luxembourg, the Netherlands, Norway, Portugal, Spain, Sweden, Switzerland, United Kingdom, Canada, the United States, Australia, New Zealand, and Japan. We considered data normalized by the EKS method [36]. We used annual GDP growth rate log(GDP_t) − log(GDP_t−1) in the clustering procedures rather than annual GDP. These series were clustered via Ward’s linkage method based on copula distance. When K = 2, the dendrogram is shown in Fig 5. The result was relatively insensitive to the maximum lag K used, with similar clustering results obtained when K ranged from two to nine.

In practice, we do not know how many distinct populations generate n time series. In general, as in any cluster analysis, the optimal number of clusters can be chosen according to some objective criterion, such as the average silhouette criterion. The average silhouette of the data is a useful criterion for assessing the natural number of clusters, which can be determined by maximizing the coefficient. It is a measure of how tightly grouped all data in the cluster are. The average silhouette coefficients were examined for different numbers of clusters, and two clusters appeared to yield a compact solution (Fig 6).

Fig 7 shows the grouping of the two clusters. In this figure, the two colors represent two groups. To display the clustering effect of copula distance, the area occupied by Europe is magnified many times. It is interesting to note that the countries were grouped primarily by geographical location. The group in blue contains five northern European countries and nearly all developed non-European countries except Japan. The northern European countries have a long history of cooperation and so have much in common. The second group is in red, and includes central European countries, southern European countries, and western European countries to the south.

Case 2: Population growth data analysis

This application considers the annual population series of 20 US states, and is aimed at identifying similarities among the population growth trend. This data is obtained from the U.S Census Bureau, Population Distribution Division (https://www.census.gov/programs-surveys/popest/data/data-sets.2007.html). It is a collection of time series of the population estimates from 1991 to 2010 in 20 states of the US, and has been used in [10]. In [10], two different groups of time series in the dataset were identified. Group 1 consisting of CA, CO, FL, GA, MD, NC, SC, TN, TX, VA, and WA had an exponentially growth trend, while Group 2 consisting of IL, MA, MI, NJ, NY, OK, PA, ND, and SD had a stable trend. In this case, we assume the above finding is the ground truth. In the following analysis, we use the growth series by taking log difference of the original time series. We cluster the data with hierarchical clustering algorithm using Ward’s linkage method. The dendrogram clustered by copula distance with Ward’s linkage method are shown in Fig 8. As we know, in this application the optimal number of clusters is 2. We can get the exact same optimal cluster number by maximizing the average silhouette width (see Fig 9). The copula distance method has the similarity index 0.8. If we section the dendrogram at the highest level we can obtain two groups, and all of the states are correctly classified except three states, CA, FL and MD.

Appendix A: Proof of Theorem 1

For every 1 ≤ i ≤ n, denote by F_i(x) the marginal distribution function of X_it. Then we will prove the theorem by the following four steps.

1. Step 1. For arbitrary u, v ∈ [0, 1], define In this step, we will prove that as T_i → ∞, .
   Specifically, for 1 ≤ i ≤ n and 1 ≤ t ≤ T_i − h, let Y_it = I(F_i(X_it) ≤ u, F_i(X_i(t+h)) ≤ v). Because X_i = (X_it, 1 ≤ t ≤ T_i) is a strictly stationary α mixing processe, one can see that Y_i = {Y_it, 1 ≤ t ≤ T_i} is also a strictly stationary α mixing processe. Then by Proposition 2.8 of [37], one can see that as T_i → ∞,
2. Step 2. In this step, we will prove that as T_i → ∞,
   Because X_i = (X_it, 1 ≤ t ≤ T_i) is a strictly stationary α mixing processe, X_i is ergodic. Then by the theorem of [38], the Glivenko-Cantelli theorem holds for time series X_i, hence we have that as T_i → ∞,
3. Step 3. In this step, we will prove that . Define By the result of Step 2, we know that P(A_i) = 1. Consequently, it suffices to prove on the event A_i.
   Specifically, On the event A_i, for an arbitrary ϵ, there exists a T such that for ∀x ∈ (−∞, ∞). Hence we have that and . Consequently, and This means that . Then let T_i → ∞, by the result of Step 1, with probability 1 we have Finally, by the arbitrary property of ϵ, one can obtain that as T_i → ∞,
4. Step 4. In this step, we will prove that as T_i, T_i′ → ∞. By the results of Step 3, we have as T_i, T_i′ → ∞, and , which means that . Moreover, , then by the dominated convergence theorem, one can obtain that as T_i, T_i′ → ∞.

This completes the whole proof of Theorem 1.

Appendix B: Proof of Proposition 1

By the Eq (2), we have By the defination of , one can see that Similarly, one can also verify that . Hence we have that

This completes the proof of Proposition 1.

Appendix C: Proof of Theorem 2

By the results of Theorem 1, we have that as min_i{T_i} → ∞, for all i, i′ = 1, ⋯, n. Hence one can prove the Theorem on the event , which is denoted by .

Assume that X_i and X_i′ belong to the j-th cluster and the j′-th cluster respectively, then we have that D_h(i, i′) = D_0,h(j, j′). One can see that D_0,h(j, j′) = 0 for j = j′, and D_0,h(j, j′) ≥ ϵ for j ≠ j′. Consequently, on the event , as long as min_i{T_i} is large enough we have if X_i, X_i′ belong to a common cluster, and if X_i, X_i′ belong to different clusters. Let , then must belong to the common cluster, and for J = n in the algorithm, are merged together as a cluster. This means that for J = n, the theorem holds.

Next assume the theorem holds for J = J₁ + 1 > J₀ + 1, we will prove the theorem still holds for J = J₁. For J = J₁ + 1, we denote the J₁ clusters as . Here one should define the dissimilarities among clusters. Without loss of generality, we only consider the single linkage method here and the proof for other dissimilarities among clusters is similar. For single linkage, the distance between is defined as . Based on the assumption that the theorem holds for J = J₁ + 1 > J₀, we know that time series in each share a common copula function. Then we have that if belong a common cluster, and otherwise. Furthermore, due to J₁ > J₀ there exist at least two , time series in which also share a common coplua function. Let . For J = J₁, and will be merged togother as a new cluster. Moreover, by the above analysis, we know and share a common copula function, and they are truely belong to a common cluster. This means that the claims of Theoren 2 is ture for J = J₁.

This completes the whole proof of Theorem 2.