Comparing recurrent networks for time-series forecasting

Recurrent Neural Networks (RNNs) have become competitive forecasting methods, as most notably shown in the winning method of the recent M4 competition. However, established statistical models such as exponential smoothing (ETS) and the autoregressive integrated moving average (ARIMA) gain their popularity not only from their high accuracy, but also because they are suitable for non-expert users in that they are robust, efficient, and automatic. In these areas, RNNs have still a long way to go. We present an extensive empirical study and an open-source software framework of existing RNN architectures for forecasting, and we develop guidelines and best practices for their use. For example, we conclude that RNNs are capable of modelling seasonality directly if the series in the dataset possess homogeneous seasonal patterns; otherwise, we recommend a deseasonalisation step. Comparisons against ETS and ARIMA demonstrate that (semi-) automatic RNN models are not silver bullets, but they are nevertheless competitive alternatives in many situations.

Informs Journal on Computing, 1993

We have studied neural networks as models for time series forecasting, and our research compares the Box-Jenkins method against the neural network method for long and short term memory series. Our work was inspired by previously published works that yielded inconsistent results about comparative performance. We have since experimented with 16 time series of di ering complexity using neural networks. The performance of the neural networks is compared with that of the Box-Jenkins method. Our experiments indicate that for time series with long memory, both methods produced comparable results. However, for series with short memory, neural networks outperformed the Box-Jenkins model. Because neural networks can be easily built for multiple-step-ahead forecasting, they present a better long term forecast model than the Box-Jenkins method. We discussed the representation ability, the model building process and the applicability of the neural net approach. Neural networks appear to provide a promising alternative for time series forecasting.

This study investigates the potential of Time Lag Recurrent Neural Networks (TLRN) for modeling the daily inflow into Eleviyan reservoir, Iran. TLRN are extended with short term memory structures that have local recurrent connections, thus making them an appropriate model for processing temporal (time-varying) information. For this study, the daily inflow into Eleviyan reservoir between years 2004 to 2007 was considered. To compare the performance of TLRN, a back propagation neural network was used. The TLRN model with gamma memory structure, eight input layer nodes, two hidden layer and one output layer (8-2-1) was found performing best out of three different models used in forecasting daily inflow. A comparison of results with back propagation neural network suggest that neither TLRN nor back propagation approaches were good in forecasting high inflow but, both approaches perform well when used to forecast low inflow values. However, statistical test suggests that both TLRN and back propagation neural network models were able to reproduce similar basic statistics as that of the actual data.

SSRN Electronic Journal, 2000

Reservoir computing is a recently introduced machine learning paradigm that has already shown excellent performances in the processing of empirical data. We study a particular kind of reservoir computers called time-delay reservoirs that are constructed out of the sampling of the solution of a time-delay differential equation and show their good performance in the forecasting of the conditional covariances associated to multivariate discrete-time nonlinear stochastic processes of VEC-GARCH type as well as in the prediction of factual daily market realized volatilities computed with intraday quotes, using as training input daily log-return series of moderate size. We tackle some problems associated to the lack of task-universality for individually operating reservoirs and propose a solution based on the use of parallel arrays of time-delay reservoirs.

Proceedings of the International Joint Conference on Neural Networks, 2011

Evolutionary algorithms are very efficient at finding "optimal" solutions for a variety of problems because they do not impose many limitations encountered in traditional methods. Reservoir Computing is a type of recurrent neural network that allows for the black box modeling of (nonlinear) dynamic systems. In contrast to other recurrent neural network approaches, Reservoir Computing does not train the input and internal weights of the network; only the output layer is trained. However, it is necessary to adjust parameters and topology to create a "good" reservoir for a given application. This study compares three different evolutionary methods in order to find the best reservoir applied to the task of time series forecasting. The results obtained with the methods are compared regarding the performance (prediction error) and regarding the computational complexity (time). We used three sets to compare the methods' results. The results show that it is possible to find well-adjusted networks automatically and that the weights search, without restriction of the echo state property, allows for more adequate solutions to be found for the problem with a lower computational cost.

Computación y …, 2011

In this work a feed-forward NN based NAR model for forecasting time series is presented. The learning rule used to adjust the NN weights is based on the Levenberg-Marquardt method. In function of the long or short term stochastic dependence of the time series, we propose an online heuristic law to set the training process and to modify the NN topology. The approach is tested over five time series obtained from samples of the Mackey-Glass delay differential equations and from monthly cumulative rainfall. Three sets of parameters for MG solution were used, whereas the monthly cumulative rainfall belongs to two different sites and times period, La Perla 1962-1971 and Santa Francisca 200-2010, both located at Córdoba, Argentina. The approach performance presented is shown by forecasting the 18 future values from each time series simulated by a Monte Carlo of 500 trials with fractional Gaussian noise to specify the variance.

2006

Time series forecasting is an active area for the application of Artificial Neural Networks (ANNs). Although the selection of an ANN has been greatly simplified, it remains a challenge to adequately determine the ANN's parameters. In this work a method based on statistical analysis and optimization techniques is proposed to select the ANNs parameters for application in time series forecasting.

2002

Artificial neural networks (ANN) have received a great deal of attention in many fields of engineering and science. Inspired by the study of brain architecture, ANN represent a class of non-linear models capable of learning from data. ANN have been applied in many areas where statistical methods are traditionally employed. They have been used in pattern recognition, classification, prediction and process control. The purpose of this paper is to discuss ANN and compare them to non-linear time series models. We begin exploring recent developments in time series forecasting with particular emphasis on the use of non-linear models. Thereafter we include a review of recent results on the topic of ANN. The relevance of ANN models for the statistical methods is considered using time series prediction problems. Finally we construct asymptotic prediction intervals for ANN and show how to use prediction intervals to choose the number of nodes in the ANN.

Bulletin of Electrical Engineering and Informatics, 2021

This study aims to determine an automatic forecasting method of univariate time series, using the nonlinear autoregressive neural network model with exogenous input (NARX). In this automatic setting, users only need to supply the input of time series. Then, an automatic forecasting algorithm sets up the appropriate features, estimate the parameters in the model, and calculate forecasts, without the users' intervention. The algorithm method used include preprocessing, tests for trends, and the application of first differences. The time series were tested for seasonality, and seasonal differences were obtained from a successful analysis. These series were also linearly scaled to [−1, +1]. The autoregressive lags and hidden neurons were further selected through the stepwise and optimization algorithms, respectively. The 20 NARX models were fitted with different random starting weights, and the forecasts were combined using the ensemble operator, in order to obtain the final product. This proposed method was applied to real data, and its performance was compared with several available automatic models in the literature. The forecasting accuracy was also measured by mean squared error (MSE) and mean absolute percent error (MAPE), and the results showed that the proposed method outperformed the other automatic models. This is an open access article under the CC BY-SA license.

Uses a parametric statistical framework to understand the effect of input representation on performance for nonlinear prediction of time series. In particular, considerations of input representation lead directly to choices between feedforward and recurrent neural networks. It is shown that feedforward networks are nonlinear autoregressive models and that recurrent networks can model a larger class of processes, including nonlinear autoregressive moving average models. For some processes, feedback allows recurrent networks to achieve better predictions than can be made with a feedforward network with a finite number of inputs. The results are confirmed on a problem in power system regional load forecasting