https://orcid.org/0000-0001-7476-3201
Figures
Abstract
This study aims to investigate whether combining forecasts generated from different models can improve forecast accuracy rather than individual models using the hierarchical time series. Various approaches of hierarchical forecasting have been considered; a bottom-up, top-down, and an optimal combination approach. Autoregressive moving averages (ARIMA) and exponential smoothing (ETS) were used as forecasting models in creating forecasting for all levels in the hierarchy to show the effect of different forecasting methods for each hierarchical model. The results indicated that the Minimum Trace Sample estimator (MinT-Sample) and the bottom-up approaches with the ARIMA model have good predictive performance than other approaches. Moreover, the forecasts from the MinT-Sample and bottom-up approaches were combined using five different combining methods. The experimental results showed that the (AC) method is superior to all other combining methods and more accurate than other individual models at level zero (international total trade in Egypt) and level one (total exports, and total imports). So, combining forecasts generated from different models by hierarchical time series leads to more accurate forecasting of the value of imports and exports which will improve the overall international trade performance, and that is through using the forecasting values of imports and exports to plan for improving the trade balance and drawing up a more efficient production policy. Finally, the study recommends using hierarchical forecasting methods in the areas of international trade, and the Ministry of Commerce and Industry could adopt the results of this study to produce precise forecasts for international trade. Moreover, the results of this study are to be a guide for the researchers to apply these approaches in other fields to improve the performance of forecasting.
Citation: A. H. Mohamed R (2023) Enhancing forecast accuracy using combination methods for the hierarchical time series approach. PLoS ONE 18(7): e0287897. https://doi.org/10.1371/journal.pone.0287897
Editor: Sani Isah Abba, King Fahd University of Petroleum and Minerals, SAUDI ARABIA
Received: February 13, 2023; Accepted: June 15, 2023; Published: July 17, 2023
Copyright: © 2023 Rania A. H. Mohamed. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are available to everyone from the Central Agency for Public Mobilization and Statistics (EGYPT): https://www.capmas.gov.eg/Pages/Publications.aspx?page_id=5107&Year=23614.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
Hierarchical time series, are multiple time series that are organized hierarchically and can be grouped at many different levels into groups based on geographic location, products, or other features. There are many specialized strategies, such as bottom-up, top-down, or a combination of the two, called “middle-out” and the optimal combination approach, [1]. In [2], Hyndman et al. propose an approach to hierarchical forecasting which provides optimal forecasts that are better than forecasts produced by either a top-down or a bottom-up approach. The method is based on independently forecasting all series at all levels of the hierarchy and then using a regression model to optimally combine and reconcile these forecasts. The resulting revised forecasts add up appropriately across the hierarchy, are unbiased, and have minimum variance amongst all combination forecasts under some simple assumptions. The simulation study shows that the method performs well compared to the top-down approach and the bottom-up method, the proposed method demonstrates by forecasting Australian tourism demand where the data are disaggregated by the purpose of travel and geographical region. In [3], Makoni and Chikobvu presented a paper that aims to model and forecast the Victoria Falls Rainforest tourism demand using hierarchical forecasting methods, the top-down, bottom-up, and optimal combination approaches are adopted. The exponential smoothing techniques (EST) and the autoregressive integrated moving average (ARIMA) methods are the forecasting methods considered. Accuracy measures indicated the bottom-up approach under ARIMA models as the best approach to the data and produced sensible future tourism forecasts. Oliveira and Ramos [4] investigate the relative performance of independent and reconciled forecasting approaches, using real data from a Portuguese retailer, two alternative forecasting model families for generating the base forecasts are considered; namely, state space models and ARIMA. Appropriate models from both families are chosen for each time series by minimizing the bias-corrected Akaike information criteria. The results show significant improvements in forecast accuracy, providing valuable information to support management decisions. And it is clear that reconciled forecasts using the Minimum Trace Shrinkage estimator (MinT-Shrink) generally improve on the accuracy of the ARIMA base forecasts for all levels and for the complete hierarchy, across all forecast horizons. Rehman and et al. [5], show that hierarchical time series arise in manufacturing and service industries when the products or services have a hierarchical structure, and top-down and bottom-up methods are commonly used to forecast the hierarchical time series, one of the critical factors that affect the performance of the two methods is the correlation between the data series, which this study attempts to resolve this problem and shows that the top-down method performs better when data have a high positive correlation compared to a high negative correlation and a combination of forecasting methods may be the best solution when there is no evidence of the correlation ship. The results show that the regression-based, VAR-COV, and Rank-based methods perform better compared to the other methods. Silveira and Azevedo [6], analyzed hourly power generation in Brazil (2018–2020), grouped according to each of the electrical subsystems and their respective sources of generating energy. The objective was to calculate the accuracy of the main measures of aggregating and disaggregating the forecasts of the Autoregressive Integrated Moving Average and Error, Trend, and Seasonal models (ETS). Specifically, the hierarchical approaches were analyzed: bottom-up, top-down, and optimal reconciliation. The optimal reconciliation models showed the best mean performance, considering the primary predictive windows. It was also found that energy forecasts in the South subsystem presented greater inaccuracy compared to the others, which signals the need for individualized models for this subsystem. Makoni and et al. [7], presented a paper whose objectives are as follows: First, to adopt the hierarchical forecasting methods in modeling and forecasting international tourist arrivals in Zimbabwe; Second, to come up with Zimbabwe international tourist arrivals Prediction Intervals (PIs) in Quantile Regression Averaging (QRA) to hierarchical tourism forecasts. Zimbabwe’s monthly international tourist arrivals data from January 2002 to December 2018 was used. The data set was disaggregated according to the purpose of the visit. Three hierarchical forecasting approaches, top-down, bottom-up, and optimal combination approaches were applied to the data. The results showed the superiority of the bottom-up approach over both the top-down and optimal combination approaches. Forecasts indicate a general increase in aggregate series. The combined methods provide new insight into modeling tourist arrivals.
This study presents a new main motivation for conducting to investigate whether combining forecasts generated from different hierarchical time series models can improve forecast accuracy rather than using individual models.
In summary, the contributions to the study are:
- Using hierarchical time series through three different approaches; the bottom-up approach, the top-down approach, and the optimal combination approach
- Combining the forecasts from the best hierarchical forecasting approaches using five combination methods, the Simple Average method, Geometric Mean, Variance—Covariance method, AKAIKE Weights, and AC method.
- Comparing the performance of the individual forecasts to the combination forecasts using Mean Absolute Percentage Error (MAPE), and Root Means Error (RMSE) as the most popular forecast error measures.
- Providing new insight into modeling international trade in Egypt that benefits the government, exporters, and importers.
This study is organized as follows. Section 2 provided a brief description of the methodology. The data is provided in Section 3, and the results are presented in Section 4. Finally, the conclusions are summarized.
2 Methodology
There are many applications in the field of business and economics that are organized hierarchically and can be grouped at several different levels into groups based on geography, products, or some other features, which are called hierarchical time series such as international trade data. There are common approaches are using for forecasting hierarchical time series; bottom-up, top-down, and Optimal combination approach proposed by Hyndman et al. [2] which has many advantages; Presents point forecasts that are reconciled across the levels of the hierarchy; allows for the interactions and correlations between the series at every level, Presents estimates of forecast uncertainty which are reconciled across the levels; the approach is flexible and provides optimal forecasts under some simple assumptions. The hierarchical forecasting approaches can capture the changes in international trade data and generate accurate forecasts. This section is divided into three parts, the first one briefly introduced the hierarchical approaches used in this study, the second part introduced the forecasting models (ARIMA and ETS) and the third part presented some combining methods.
2.1 Hierarchical forecasting approaches
Three approaches are used in this study: bottom-up, top-down, and optimal combination.
2.1.1 The bottom-up approach.
One of the most common approaches used for hierarchical forecasting is the bottom-up approach, which requires first providing forecasts for each series at the bottom level, and then aggregating them to provide forecasts for all the levels of the hierarchal structure. The advantage of this approach is that, by modeling the data at the most disaggregated bottom level, no information is lost due to aggregation. Therefore, the dynamics of the individual series can be better captured. However, bottom-level data can be quite noisy, and, therefore, more challenging to model.
The hierarchical methods can be represented by the general form:
(1)
Where S is the m × mk summing matrix, and P is a Matrix of order mk × m, the role of P changes depending on the hierarchical approach. To represent the bottom-up approach using Eq (1),
where 0i×j is the i × j null matrix, the role of P is to extract the bottom-level forecasts, which are subsequently aggregated by the summation matrix S to provide the revised forecasts for the whole hierarchy. For more detail, refer to [6, 8].
2.1.2 Top-down approach.
The top-down approach is the other common approach in hierarchical forecasting; the approach disaggregates the forecasts of the total series and distributes these down the hierarchy depending on historical data proportions, to represent the top-down approach using Eq (1),
Where p = [p1, p2, …, pmk]′ is a set of proportions for the bottom-level series, the role of P is to distribute the top-level forecasts to forecasts for the bottom-level series. Different top-down forecasting methods lead to other proportionality vectors p.
In this study, three models of this approach are used [8].
- Top-down forecasts based on the average historical proportions (Gross-Sohl method A) (TDGSA)
Each proportion pj reflects the average of the historical proportions of the bottom level series Yj,t over the period t = 1, …, n, relative to the total aggregate Yt
- Top-down forecasts based on the proportion of historical averages (Gross-Sohl method F) (TDGSF)
Each proportion pj captures the average historical value of the bottom level series Yj,t relative to the average value of the total aggregate Yt.
- Top-down forecasts using forecast proportions (TDFP)
where
is the h-step-ahead forecast and
is the sum of the h-step-ahead forecasts below node j which are directly connected to node j. For more detail, refer to [6, 8, 9].
2.1.3 Optimal combination approach.
Hyndman et al. [2] suggested a new approach to forecasting hierarchical models; the method is based on independently forecasting all series at all levels of the hierarchy and then combining and reconciling these forecasts using a regression model. The base forecasts can be written as follows:
(2)
Where βn(h) = E[YK,n+h ∣ Y1, …, Yn] is the unknown mean of the bottom level K, εh has zero mean, and covariance matrix Var(εh) = Σh, then estimate βn(h) by treating Eq (2) as a regression equation and obtain forecasts for all levels of the hierarchy. If Σh was known, generalized least squares estimation is used to get the minimum variance unbiased estimate of βn(h) as:
Where
is the Moore-Penrose generalized inverse of Σh, the revised forecasts is given by:
(3)
Where
; which satisfies the unbiasedness property SPS = S. This condition is valid for the bottom-up approach, although not for the top-down approach because of SPS ≠ S. So, the top-down approaches will never give unbiased forecasts, even if the base forecasts are unbiased. The variance of these forecasts is given by:
In general, Σh is not known and is not identifiable [10]. The residuals from the regression model in Eq (2) are given by
To identify Σh, (Im − SP) must be invertible and therefore of full rank. But SPS = S implies that S′SPS = S′S, where S′S is positive definite, So, PS = Imk.
Then, Rank(Im − SP) = tr(Im − SP) = tr(Im) − tr(Imk) = m − mk, and (Im − SP) is rank deficient; consequently, Σh cannot be identified. The covariance matrix of the h-step ahead reconciled forecast errors is:
For any P such that SPS = S, where is the variance-covariance matrix of the h-step ahead base forecast errors and is given by Eq (3). The purpose is to find the matrix P that minimizes the trace of Var
satisfying SPS = S, which gives the best (minimum variance) linear unbiased reconciled forecasts, which refer to this as MinT(minimum trace) reconciliation which computed as
Where Wh is the covariance matrix of the base forecast errors. Although Wh does not suffer from a lack of identification, it is difficult to estimate, especially for h > 1. There are several methods to estimate Wh such as [10]:
OLS: Wh = khI, ∀h, where kh > 0 [2]. This method is optimal when the base forecast errors are uncorrelated and equivalent.
WLSv: , where kh > 0 and
.
In this case, MinT is described as a weighted least squares (WLS) estimator applying variance scaling.
WLSs: Wh = khΛ, ∀h, where kh > 0 and Λ = diag(S1) with 1 being a unit column vector of dimension n.
MinT(Sample): , where kh > 0 the unrestricted sample covariance estimator for h = 1.
MinT(Shrink): , where
, is a shrinkage estimator with diagonal target,
is a diagonal matrix comprising the diagonal entries of
, and λD is the shrinkage intensity parameter. For more detail, refer to [2, 9, 10].
2.2 Forecasting models
Two forecasting models were presented briefly in this subsection for generating forecasts for all levels in the hierarchy to show the influence of different forecasting methods for each hierarchical model: ARIMA and ETS models. They are based on different perspectives on the problem and often, but not always, perform differently, although they share some mathematically equivalent models (Oliveira and Ramos [4]).
2.2.1 ARIMA models.
ARIMA models are one of the most popular models for forecasting time series, different types of stochastic seasonal and non-seasonal time series can be represented by these models. The general form of seasonal ARIMA models is:
Where: ϕp(β) and θq(β) are the regular autoregressive and moving average polynomials of orders p and q, respectively, ΦP (βS) and θQ (βS) are the seasonal autoregressive and moving average polynomials of orders P and Q, respectively, S is the period of seasonality, D is the degree of seasonal differencing is the degree of ordinary differencing, B is backshift operator; εt is a white noise process with zero mean and variance σ2 [4]. For more detail, refer to [11].
2.2.2 Exponential smoothing.
Ord et al. [12] extended the work of Snyder [13] by proposing a class of innovation state-space models which considered as underlying some of the exponential smoothing methods. Hyndman et al. [14] and Taylor [15] extended this to include fifteen exponential smoothing methods [16]. These fifteen methods are discriminated based on the nature of the trend and seasonality component observed [17]. Hyndman et al. [18, 19] described two possible innovations in state space models for each of the fifteen models, resulting in thirty different models, they have developed an automatic forecasting method using these models, a triplet (E, T, S) was used. E, T, S stands for error, trend, and seasonality components, respectively. The general model involves a state vector xt = (lt, bt, st, st−1, …, st−m+1)′ and state-space equations have
where εt is a Gaussian white noise process with mean zero and variance σ2, μt = w(xt−1). There are two state-space models: one model with additive errors and the other with multiplicative errors. The model with additive errors has r(xt−1) = 1, so
The model with multiplicative errors has r(xt−1) = μt, so:
Thus,
is the relative error for the multiplicative model and any value of r(xt−1) will lead to identical point forecast for yt [17]. For more detail, refer to [14, 18, 19].
2.3 Combined forecasts
Combined forecasts were introduced by Bates and Granger [20], and several forecast combination methods have been developed in the literature [21, 22]. In this study, five combination methods were used: The Simple Average method, Geometric Mean, Variance—Covariance method, AKAIKE Weights, and AC method.
2.3.1 Simple Average (SA).
In this method, the forecasts are combined by assigning equal weights to each individual forecast. The combination forecast can be expressed as
Where fi is the ith single forecast; fc is the combined forecast generated by the n single forecast fi; n is the total number of individual forecasting models, based on Wong et al. [23, 24], and wi is the combination weight assigned to fi which is specified as
.
2.3.2 Geometric Mean (GM).
Suppose the combined forecasts from two forecasting models, the combined forecast using Geometric Mean can be expressed as:
Where fc is the combined forecast based on the individual forecasts of f1t and f2t.
2.3.3 Variance-Covariance (VACO).
This method was proposed by Bates and Grager [20]. Due to Shen et al. [25, 26], Suppose the combined forecasts from two unbiased forecasting models are given as
where fc is the combined forecast based on the individual forecasts of f1t and f2t, w and (1 − w) are the weights assigned to f1t and f2t, respectively. The weight that minimizes the combined forecast variance is:
(4)
where
and
are the unconditional individual forecast error variance and σ12 is the covariance. In practical, Bates and Granger [20] suggested Eq (4) to combine the forecasts
(5)
Where e1t and e2t are individual forecast errors, and T is the sample size. For more than two individual forecasts the weights can be calculated, according to Fritz et al. [27] by
2.3.4 AKAIKE weights.
In this method, Akaike’s Information Criterion (AIC) is computed for each model, (Burnham and Anderson [28]; Acquah [29]; Hsiao and Wan [30]; Pi latowska [31]) and the weights can be calculated as follows
Where AICmin is the minimum of the N different AICi values.
2.3.5 AC.
Altavilla and Ciccarelli [32] Proposed a new methodology that modifies (Aggregated Forecast Through Exponential Reweighting (AFTER) proposed by Yang [33], for more detail, refer to [33, 34]). the weight attributed to a certain model at time τ is larger the larger its ability to forecast the actual value not in all previous periods, but only at τ − 1.
The weights are assigned as follows:
Where:
the sample variance of dependent variable,
, and
3 Data
The hierarchical structure of the dataset was illustrated in Fig 1, and the number of series at each hierarchical level was presented in Table 1.
Table 1 concludes that the hierarchy consists of three levels, the top-level (level 0) represents the international total trade, the middle level (level 1) total exports, and total imports separately, and the bottom level (level 2) shows the way disaggregated. Fig 2 shows the characteristics of the disaggregated series. Thus, the hierarchy includes 7-time series, each containing 70 monthly observations from January 2015 to October 2020 as an estimation period and the data from November 2020 to April 2021 as a testing period, the data were obtained from the Central Agency for Public Mobilization and Statistics (EGYPT). https://www.capmas.gov.eg/Pages/Publications.aspx?page_id=5107&Year=23614.
According to Fig 2, non-Petroleum Imports constitute a larger percentage of the total imports, also non-Petroleum Exports constitute a larger percentage of the total Exports. Generally, there is a Fluctuation as shown in Fig 2.
4 The results
This section provides the empirical results obtained, starting with presenting the descriptive statistics of the data, then comparing the performance between hierarchical approaches, and finally comparing the performance of the individual forecasts with the combined forecasts.
4.1 Descriptive statistics
The descriptive statistics of the series are presented in Table 2. Which indicates that the mean of the BA (Non-Petroleum Imports) is the highest, the minimum is 121 for Export AB (Crude Oil & its Products), and the data is skewed to the left as indicated by the skewness value except AB and BB (Crude Oil & its Products). Kurtosis values indicated that most of the data are leptokurtic.
4.2 Results from individual models
Mean Absolute Percentage Error (MAPE), and Root Means Error (RMSE) were used as the most popular forecast error measures. MAPE is a relative measure of performance, which specifically was used frequently in the literature studies of hierarchical time series. The MAPE was used in a comparison with the RMSE measure:
Where
are the estimated and actual values respectively, n is the number of data (Silveira and Azevedo [6]).
Tables 3–6 contain the values of the MAPE and RMSE for each forecast horizon yielded by bottom-up, top-down, and the optimal combination approach as described in section 2.1, and the last column contains the average across all forecast horizons for ETS and ARIMA models using R software.
Tables 3–6 concluded that the error percentage produced by the ARIMA model was less than that produced by the ETS model at all levels, and it is clear that the Minimum Trace Sample estimator and the bottom-up approach with ARIMA models have good predictive performance, even with the increased in the forecast horizon. The MinT(shrink) approach is better than optimal combination approaches such as OLS and WLS under the ARIMA model. Moreover, the results obtained from the top-down approach did not present good predictive results.
4.3 Results from combining forecasts
To investigate empirically whether combining the forecasts generated from different models can improve forecasting accuracy rather than using individual models is the main focus of this study. So, The forecasts from the best individual hierarchical forecasting approaches were combined (MinT -Sample and the bottom-up) using combination methods as mentioned in section 2.3, and the performance of comparison was evaluated based on the MAPE and RMSE. (See Tables 7 and 8).
The results of this study showed that the (AC) method works better than other combination methods for forecasting the value of imports and exports, and it is more accurate than the best individual forecasts at level zero (international total trade in Egypt) and level one (total exports and total imports), which leads to providing forecasting values that can be economically relied upon in planning international trade.
5 Conclusion
This study used different hierarchical forecasting approaches to obtain the best models and forecasts for international trade in Egypt. ARIMA and Exponential Smoothing (ETS) were used as forecasting models for all levels in the hierarchy. The results concluded that the error rate generated by the ARIMA model was lower than that generated by the ETS model. The minimum trace sample estimator and the bottom-up approaches with ARIMA models have good forecasting performance compared to the other approaches. Furthermore, the forecasts from the Minimum Trace sample estimator and bottom-up approaches with ARIMA models were combined as the best individual approaches using five different combining methods. The results indicated that the (AC) method performs better than other combination methods for forecasting the value of imports and exports, and it is more accurate than the best individual forecasting approaches at level zero (international total trade in Egypt) and level one (total exports and total imports). So, the (AC) method leads to providing forecasting values more accurate than can be economically relied upon in improving the trade balance and drawing up a more efficient production policy. Therefore, the combined forecasts provide a new insight into modeling international trade in Egypt that benefits the government, exporters, and importers. The research recommends using hierarchical forecasting methods in the areas of international trade volume because they produce acceptable forecasts.
References
- 1. Mitra D, Paul R K, Pal S. Hierarchical time-series models for forecasting oilseeds and pulses production in India. Economic Affairs. 2017;62(1):103–111.
- 2. Hyndman R J, Ahmed R A, Athanasopoulos G, Shang HL. Optimal combination forecasts for hierarchical time series. Computational statistics & data analysis. 2011 Sep 1;55(9):2579–2589.
- 3. Makoni T, Chikobvu D. Hierarchical forecasting of tourist arrivals at the Victoria falls rainforest, Zimbabwe. African Journal of Hospitality, Tourism and Leisure. 2018;7(4):1–15.
- 4. Oliveira J M, Ramos P. Assessing the performance of hierarchical forecasting methods on the retail sector. Entropy. 2019 Apr 24;21(4):436. pmid:33267150
- 5. Rehman HU, Wan G, Ullah A, Shaukat B. Individual and combination approaches to forecasting hierarchical time series with correlated data: an empirical study. Journal of Management Analytics. 2019 Jul 3;6(3):231–49.
- 6. Silveira Gontijo T, Azevedo Costa M. Forecasting hierarchical time series in power generation. Energies. 2020 Jul 20;13(14):3722.
- 7. Makoni T, Chikobvu D, Sigauke C. Hierarchical forecasting of the Zimbabwe International Tourist arrivals. Statistics, Optimization & Information Computing. 2021 Jan 20;9(1):137–156.
- 8. Athanasopoulos G, Ahmed RA, Hyndman RJ. Hierarchical forecasts for Australian domestic tourism. International Journal of Forecasting. 2009 Jan 1;25(1):146–166.
- 9.
Hyndman RJ, Athanasopoulos G, Shang HL. hts: An R Package for Forecasting Hierarchical or Grouped Time Series. R Package (Available at http://cran.unej.ac.id/web/packages/hts/vignettes/hts.pdf). 2015.
- 10. Wickramasuriya SL, Athanasopoulos G, Hyndman RJ. Optimal forecast reconciliation for hierarchical and grouped time series through trace minimization. Journal of the American Statistical Association. 2019 Apr 3;114(526):804–819.
- 11.
Box G E, Jenkins G M, Reinsel G C, Ljung G M. Time series analysis: forecasting and control. John Wiley & Sons; 2015, Hoboken, New Jersey, pp. 712. ISBN: 978-1-118-67502-1
- 12. Ord JK, Koehler AB, Snyder RD. Estimation and prediction for a class of dynamic nonlinear statistical models. Journal of the American Statistical Association. 1997 Dec 1;92(440):1621–1629.
- 13. Snyder RD. Recursive estimation of dynamic linear models. Journal of the Royal Statistical Society. Series B (Methodological). 1985 Jan 1:272–276.
- 14. Hyndman R J, Koehler A B, Snyder R D, Grose S. A state space framework for automatic forecasting using exponential smoothing methods. International Journal of forecasting. 2002 Jul 1;18(3):439–454.
- 15. Taylor J W. Exponential smoothing with a damped multiplicative trend. International journal of Forecasting. 2003 Oct 1;19(4):715–725.
- 16. De Gooijer J G, Hyndman RJ. 25 years of time series forecasting. International journal of forecasting. 2006 Jan 1;22(3):443–473.
- 17. Panigrahi S, Behera HS. A hybrid ETS–ANN model for time series forecasting. Engineering applications of artificial intelligence. 2017 Nov 1;66:49–59.
- 18. Hyndman RJ, Khandakar Y. Automatic time series forecasting: the forecast package for R. Journal of statistical software. 2008 Jul 29;27:1–22.
- 19.
Hyndman R, Koehler AB, Ord JK, Snyder RD. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media; 2008 Jun 19.
- 20. Bates J M, Granger C W. The combination of forecasts. Journal of the operational research society. 1969 Dec 1;20(4):451–468.
- 21. Mohamed R H, A Model to Enhance the Forecasting Efficiency of Suez Canal Revenues Using Combining Methods, The Egyptian Statistical Journal, 2015, 59(2), pp.127–141.
- 22. Clemen R T. Combining forecasts: A review and annotated bibliography. International journal of forecasting. 1989 Jan 1;5(4):559–583.
- 23. Wong K K, Song H, Witt SF, Wu DC. Tourism forecasting: to combine or not to combine? Tourism management. 2007 Aug 1;28(4):1068–1078.
- 24. Hibon M, Evgeniou T. To combine or not to combine: selecting among forecasts and their combinations. International journal of forecasting. 2005 Jan 1;21(1):15–24.
- 25. Shen S, Li G, Song H. An assessment of combining tourism demand forecasts over different time horizons. Journal of Travel Research. 2008 Nov;47(2):197–207.
- 26. Shen S, Li G, Song H. Combination forecasts of international tourism demand. Annals of Tourism Research. 2011 Jan 1;38(1):72–89.
- 27. Fritz R G, Brandon C, Xander J. Combining time-series and econometric forecast of tourism activity. Annals of Tourism Research. 1984 Jan 1;11(2):219–229.
- 28. Burnham K P, Anderson D R. Multimodel inference: understanding AIC and BIC in model selection. Sociological methods & research. 2004 Nov;33(2):261–304.
- 29. Acquah HD. A bootstrap approach to evaluating the performance of Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) in selection of an asymmetric price relationship. Journal of Agricultural Sciences, Belgrade. 2012;57(2):99–110.
- 30. Hsiao C, Wan SK. Is there an optimal forecast combination?. Journal of Econometrics. 2014 Jan 1;178:294–309.
- 31. Pilatowska M. Combined Forecasts Using the Akaike Weights. Dynamic 345 Econometric Models. 2009, vol. 9, pages 5–16
- 32.
Altavilla C., Ciccarelli M., Information Combination and Forecast (st) Ability evidence from Vintages of Time Series Data, 2007, European Central Bank, No.846.
- 33. Yang Y. Combining forecasting procedures: some theoretical results. Econometric Theory. 2004 Feb;20(1):176–222.
- 34. Zou H, Yang Y. Combining time series models for forecasting. International journal of Forecasting. 2004 Jan 1;20(1):69–84.