Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Numerical Results
Artificial Data
Real-World Data
Index Data
Conclusion
References
All Topics
Computer Science

Neural Networks for Time-Series Forecasting

Profile image of William RemusWilliam Remus

2012, Handbook of Natural Computing

https://doi.org/10.1007/978-3-540-92910-9_14
visibility

…

description

15 pages

link

1 file

Abstract

In this paper we study the generalization capabilities of fully-connected neural networks trained in the context of time series forecasting. Time series do not satisfy the typical assumption in statistical learning theory of the data being i.i.d. samples from some data-generating distribution. We use the input and weight Hessians, that is the smoothness of the learned function with respect to the input and the width of the minimum in weight space, to quantify a network's ability to generalize to unseen data. While such generalization metrics have been studied extensively in the i.i.d. setting of for example image recognition, here we empirically validate their use in the task of time series forecasting. Furthermore we discuss how one can control the generalization capability of the network by means of the training process using the learning rate, batch size and the number of training iterations as controls. Using these hyperparameters one can efficiently control the complexity of the output function without imposing explicit constraints.

Related papers

Time Series Forecasting by Generalized Regression Neural Networks Trained With Multiple Series
Francisco José Quiroz Martínez

IEEE Access, 2022

Time series forecasting plays a key role in many fields such as business, energy or environment. Traditionally, statistical or machine learning models for time series forecasting are trained with the historical values of the series to be forecast. Unfortunately, some time series are too short to suitably train a model. Motivated by this fact, this paper explores the use of data available in a pool or collection of time series to train a model that predicts an individual series. Concretely, we train a generalized regression neural network with the examples drawn from the historical values of a pool of series and then use the model to forecast individual series. In this sense several approaches are proposed, including to draw the examples from a pool of series related to the series to be forecast or the training of several models with mutually exclusive series and the combination of their forecasts. Experimental results in terms of forecasting accuracy using generalized regression neural networks are promising. Furthermore, the proposed approaches allow to forecast series that are too short to build a traditional generalized regression neural network model. INDEX TERMS Generalized regression neural networks, model combination, time series forecasting.

View PDFchevron_right
Generalization by weight-elimination with application to forecasting
Bernardo Huberman

Nips, 1991

Inspired by the information theoretic idea of minimum description length, we add a term to the back propagation cost function that penalizes network complexity. We give the details of the procedure, called weight-elimination, describe its dynamics, and clarify the meaning of the parameters involved. From a Bayesian perspective, the complexity term can be usefully interpreted as an assumption about prior distribution of the weights. We use this procedure to predict the sunspot time series and the notoriously noisy series of currency exchange rates.

View PDFchevron_right
Statistical Characterization and Optimization of Artificial Neural Networks in Time Series Forecasting: The One Period Forecast Case
M. Angelica Salazar Aguilar

2005

View PDFchevron_right
Input-variable specification for Neural Networks - An analysis of forecasting low and high time series frequency
Sven Crone

Prior research in forecasting time series with Neural Networks (NN) has provided inconsistent evidence on their predictive accuracy. In management, NN have shown only inferior performance on well established benchmark time series of monthly, quarterly or annual frequency. In contrast, NN have shown preeminent accuracy in electrical load forecasting on daily or hourly time series, leading to successful real life applications. While this inconsistency has been traditionally attributed to the lack of a reliable methodology to model NNs, recent research indicates that the particular data properties of high frequency time series may be equally important. High frequency time series of daily, hourly or even shorter time intervals pose additional modelling challenges in the length and structure of the time series, which may abet the use of novel methods. This analysis aims to identify and contrast the challenges in modelling NN for low and high frequency data in order to develop a unifying forecasting methodology tailored to the properties of the dataset. We conduct a set of experiments in three different frequency domains of daily, weekly and monthly data of one empirical time series of cash machine withdrawals, using a consistent modelling procedure. While our analysis provides evidence that NN are suitable to predict high frequency data, it also identifies a set of challenges in modelling NN that arise from high frequency data, in particular in specifying the input vector, and that require specific modelling approaches applicable to both low and high frequency data.

View PDFchevron_right
Comparative analysis of neural networking and regression models for time series forecasting
Станислав Шолтанюк

Comparative analysis of neural networking and regression models for time series forecasting, 2019

Applicability of neural nets in time series forecasting has been considered and researched. For this, training of neural network on various time series with preliminary selection of optimal hyperparameters has been performed. Comparative analysis of received neural networking forecasting model with linear regression has been performed. Conditions, affecting on accuracy and stability of results of the neural network, have been revealed.

View PDFchevron_right
A Neural Network Approach to Time Series Forecasting
Leslie S Smith

2009

We propose a simple approach for forecasting univariate time series. The proposed algorithm is an ensemble learning technique that combines the advice from several Generalized Regression Neural Networks. We compare our algorithm with the most used algorithms on real and synthetic datasets. The proposed algorithm appears as more powerful than existing ones.

View PDFchevron_right
Numerical conditioning of neural network components in time series forecasting
Станислав Шолтанюк

БГУИР, РБ, 2020

In this paper, an issue of stability of fully connected neural network (perceptron), forecasting time series with sliding window method, is considered. A task of finding conditions, providing best results on accuracy and stability, was set. To accomplish it, training of the neural network with various sets of hyperparameters, such as amount of neurons on input and hidden layers, as well as activation function, has been performed. Training was performed on modeled data, which have been perturbed with unbiased normally distributed random variables with various variance. Such training results as number of conditioning of weight matrix between input and hidden layers, number of conditioning of activation functions, as well as statistics on retrospective forecasting absolut error, have been shown. An approach for estimation of condition number for some activation functions has been proposed. Impact of activation function on accuracy and stability of forecasting perceptron has been shown.

View PDFchevron_right
Time Series Forecasting using Artificial Neural Networks
Rajasekhar Nannapaneni

2018

Time series datasets contain a collection of observations in time and this additional temporal component differentiates time series with normal datasets. Studies of forecasting time series datasets are particularly popular as they improve financial planning, sales prediction, average rainfall forecast, internet traffic rates, business trend forecasting, weather forecast, contagious disease spread, etc. Time series forecasting facilitates the decision process in various domains such as business by predicting timely sales information thereby improving management of product manufacturing output. In medical domain, predicting patient health improves medical treatment. The environmental and physical time series forecasting such as rainfall, disasters prediction would improve the contingency plans. On an overall basis time series prediction improves the way humans live. Artificial neural networks are increasingly becoming successful in predicting the time series data through regression-ba...

View PDFchevron_right
Multivariate Statistical Analysis for Training Process Optimization in Neural Networks-Based Forecasting Models
Christian G. Quintero M.

Applied Sciences, 2021

Data forecasting is very important for electrical analysis development, transport dimensionality, marketing strategies, etc. Hence, low error levels are required. However, in some cases data have dissimilar behaviors that can vary depending on such exogenous variables as the type of day, weather conditions, and geographical area, among others. Commonly, computational intelligence techniques (e.g., artificial neural networks) are used due to their generalization capabilities. In spite of the above, they do not have a unique way to reach optimal performance. For this reason, it is necessary to analyze the data’s behavior and their statistical features in order to identify those significant factors in the training process to guarantee a better performance. In this paper is proposed an experimental method for identifying those significant factors in the forecasting model for time series data and measure their effects on the Akaike information criterion (AIC) and the Mean Absolute Percen...

View PDFchevron_right
A Homogeneous Ensemble of Artificial Neural Networks for Time Series Forecasting
Ratnadip Adhikari

structure, 2011

Enhancing the robustness and accuracy of time series forecasting models is an active area of research. Recently, Artificial Neural Networks (ANNs) have found extensive applications in many practical forecasting problems. However, the standard backpropagation ANN training algorithm has some critical issues, e.g. it has a slow convergence rate and often converges to a local minimum, the complex pattern of error surfaces, lack of proper training parameters selection methods, etc. To overcome these drawbacks, various improved training methods have been developed in literature; but, still none of them can be guaranteed as the best for all problems. In this paper, we propose a novel weighted ensemble scheme which intelligently combines multiple training algorithms to increase the ANN forecast accuracies. The weight for each training algorithm is determined from the performance of the corresponding ANN model on the validation dataset. Experimental results on four important time series depicts that our proposed technique reduces the mentioned shortcomings of individual ANN training algorithms to a great extent. Also it achieves significantly better forecast accuracies than two other popular statistical models.

View PDFchevron_right

References (38)

  1. A. Achille, S. Soatto, Emergence of invariance and disentanglement in deep representations, J. Mach. Learn. Res. 19 (2018) 1-34.
  2. A. Agarwal, J.C. Duchi, The generalization ability of online algorithms for dependent data, IEEE Trans. Inform. Theory 59 (2013) 573-587.
  3. A. Auffinger, G.B. Arous, J. Čern ý, Random matrices and complexity of spin glasses, Commun. Pure Appl. Math. 66 (2013) 165-201.
  4. S. Becker, Y. Zhang, A.A. Lee, Geometry of Energy Landscapes and the Optimizability of Deep Neural Networks, 2018 arXiv:1808.00408.
  5. J.L. Bernier, J. Ortega, E. Ros, I. Rojas, A. Prieto, A quantitative study of fault tolerance, noise immunity, and generalization ability of MLPs, Neural Comput. 12 (2000) 2941-2964.
  6. C. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), 1st ed. 2006. corr. 2nd printing ed., Springer, New York, 2007.
  7. A.J. Bray, D.S. Dean, Statistics of critical points of Gaussian fields on large-dimensional spaces, Phys. Rev. Lett. 98 (2007) 150201.
  8. P. Chaudhari, S. Soatto, Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks, in: 2018 Information Theory and Applications Workshop (ITA), IEEE, 2018, pp. 1-10.
  9. A. Choromanska, M. Henaff, M. Mathieu, G.B. Arous, Y. LeCun, The Loss Surfaces of Multilayer Networks, 2015, pp. 192-204.
  10. R. Cont, Empirical Properties of Asset Returns: Stylized Facts and Statistical Issues, 2001.
  11. Y.N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, Y. Bengio, Identifying and attacking the saddle point problem in high-dimensional non-convex optimization, Advances in Neural Information Processing Systems (2014) 2933-2941.
  12. L. Dinh, R. Pascanu, S. Bengio, Y. Bengio, Sharp Minima Can Generalize for Deep Nets, 2017 arXiv:1703.04933.
  13. G.K. Dziugaite, D.M. Roy, Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters Than Training Data, 2017 arXiv:1703.11008.
  14. Y.V. Fyodorov, I. Williams, Replica symmetry breaking condition exposed by random matrix calculation of landscape complexity, J. Stat. Phys. 129 (2007) 1081-1116.
  15. R. Geirhos, C.R. Temme, J. Rauber, H.H. Schütt, M. Bethge, F.A. Wichmann, Generalisation in humans and deep neural networks, Advances in Neural Information Processing Systems (2018) 7549-7561.
  16. S. Hochreiter, J. Schmidhuber, Flat minima, Neural Comput. 9 (1997) 1-42.
  17. S. Jastrzkebski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, A. Storkey, Three Factors Influencing Minima in SGD, 2017 arXiv:1711.04623.
  18. V. Kuznetsov, M. Mohri, Generalization bounds for time series prediction with non-stationary processes, in: International Conference on Algorithmic Learning Theory, Springer, 2014, pp. 260-274.
  19. V. Kuznetsov, M. Mohri, Generalization bounds for non-stationary mixing processes, Mach. Learn. 106 (2017) 93-117.
  20. V. Kuznetsov, M. Mohri, Theory and Algorithms for Forecasting Time Series, 2018 arXiv:1803.05814.
  21. H. Li, Z. Xu, G. Taylor, T. Goldstein, Visualizing the Loss Landscape of Neural Nets, NIPS, 2018.
  22. S. Mandt, M.D. Hoffman, D.M. Blei, Stochastic Gradient Descent as Approximate Bayesian Inference, 2017 arXiv:1704.04289.
  23. J. Martens, Deep learning via Hessian-free optimization, ICML, vol. 27 (2010) 735-742.
  24. D.J. McDonald, C.R. Shalizi, M. Schervish, Nonparametric risk bounds for time-series forecasting, J. Mach. Learn. Res. 18 (2017) 1-40.
  25. R. Novak, Y. Bahri, D.A. Abolafia, J. Pennington, J. Sohl-Dickstein, Sensitivity and Generalization in Neural Networks: An Empirical Study, 2018 arXiv:1802.08760.
  26. D.S. Park, J. Sohl-Dickstein, Q.V. Le, S.L. Smith, The Effect of Network Width on Stochastic Gradient Descent and Generalization: An Empirical Study, 2019 arXiv:1905.03776.
  27. J. Pennington, Y. Bahri, Geometry of neural network loss surfaces via random matrix theory, International Conference on Machine Learning (2017) 2798-2806.
  28. R. Reed, S. Oh, R. Marks, Regularization using jittered training data, in: International Joint Conference on Neural Networks, 1992. IJCNN, vol. 3, IEEE, 1992, pp. 147-152.
  29. S. Seong, Y. Lee, Y. Kee, D. Han, J. Kim, Towards Flatter Loss Surface Via Nonmonotonic Learning Rate Scheduling, UAI, 2018.
  30. R. Shwartz-Ziv, N. Tishby, Opening the Black Box of Deep Neural Networks Via Information, 2017 arXiv:1703.00810.
  31. L.N. Smith, N. Topin, Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates, 2017 arXiv:1708.07120.
  32. S.L. Smith, Q.V. Le, A Bayesian Perspective on Generalization and Stochastic Gradient Descent, 2018.
  33. J. Sokoli ć, R. Giryes, G. Sapiro, M.R. Rodrigues, Robust large margin deep neural networks, IEEE Trans. Signal Process. 65 (2017) 4265-4280.
  34. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR, 2014, pp. 1929-1958.
  35. N. Tishby, N. Zaslavsky, Deep learning and the information bottleneck principle, in: 2015 IEEE Information Theory Workshop (ITW), IEEE, 2015, pp. 1-5.
  36. C. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals, Understanding Deep Learning Requires Rethinking Generalization, 2016 arXiv:1611.03530.
  37. G. Zhang, B.E. Patuwo, M.Y. Hu, Forecasting with artificial neural networks: the state of the art, Int. J. Forecast. 14 (1998) 35-62.
  38. W. Zhou, V. Veitch, M. Austern, R.P. Adams, P. Orbanz, Compressibility and Generalization in Large-Scale Deep Learning, 2018 arXiv:1804.05862.

Related papers

Some issues about the generalization of neural networks for time series prediction
J.K. Vrijling

Artificial Neural Networks: Formal …, 2005

Some issues about the generalization of ANN training are investigated through experiments with several synthetic time series and real world time series. One commonly accepted view is that when the ratio of the training sample size to the number of weights is larger than 30, the overfitting will not occur. However, it is found that even with the ratio higher than 30, overfitting still exists. In cross-validated early stopping, the ratio of cross-validation data size to training data size has no significant impact on the testing error. For stationary time series, 10% may be a practical choice. Both Bayesian regularization method and the cross-validated early stopping method are helpful when the ratio of training sample size to the number of weights is less than 20. However, the performance of early stopping is highly variable. Bayesian method outperforms the early stopping method in most cases, and in some cases even outperforms no-stop training when the training data set is large.

View PDFchevron_right
A New Methodology for Neural Network Training Ensures Error Reduction in Time Series Forecasting
Paola Sánchez

Journal of Computer Science

Artificial Neural Networks (ANN) consists of some components, such as architecture and learning algorithm. These components have a significant effect on the performance of the ANN, but finding good parameters is a difficult task to achieve. An important requirement for this task is to ensure the reduction of error when inputs and/or hidden neurons are added. In practice, it is assumed that this requirement is always true, but usually it is false. In this paper, we propose a new algorithm that ensures error decrease when input variables and/or hidden neurons are added to the neural network. The behavior of two traditional algorithms and the proposed algorithm in the forecast of Airline time series were compared. The empirical results indicate that the proposed algorithm allows a steady decrease of fit error in all cases, where de most important and differentiable feature is the fact that reach values close to zero, which is not true for the other algorithms. Therefore, it can be used as a suitable alternative algorithm, especially when it needs a good fit.

View PDFchevron_right
Feedforward Neural Nets as Models for Time Series Forecasting
Paul Fishwick

Informs Journal on Computing, 1993

We have studied neural networks as models for time series forecasting, and our research compares the Box-Jenkins method against the neural network method for long and short term memory series. Our work was inspired by previously published works that yielded inconsistent results about comparative performance. We have since experimented with 16 time series of di ering complexity using neural networks. The performance of the neural networks is compared with that of the Box-Jenkins method. Our experiments indicate that for time series with long memory, both methods produced comparable results. However, for series with short memory, neural networks outperformed the Box-Jenkins model. Because neural networks can be easily built for multiple-step-ahead forecasting, they present a better long term forecast model than the Box-Jenkins method. We discussed the representation ability, the model building process and the applicability of the neural net approach. Neural networks appear to provide a promising alternative for time series forecasting.

View PDFchevron_right
Artificial neural networks in time series forecasting: a comparative analysis
Claudio Moraga

2002

Artificial neural networks (ANN) have received a great deal of attention in many fields of engineering and science. Inspired by the study of brain architecture, ANN represent a class of non-linear models capable of learning from data. ANN have been applied in many areas where statistical methods are traditionally employed. They have been used in pattern recognition, classification, prediction and process control. The purpose of this paper is to discuss ANN and compare them to non-linear time series models. We begin exploring recent developments in time series forecasting with particular emphasis on the use of non-linear models. Thereafter we include a review of recent results on the topic of ANN. The relevance of ANN models for the statistical methods is considered using time series prediction problems. Finally we construct asymptotic prediction intervals for ANN and show how to use prediction intervals to choose the number of nodes in the ANN.

View PDFchevron_right
Time Series Prediction and Neural Networks
NANDIPATI UDAY SAI JAGANNADH

Neural Network approaches to time series prediction are briefly discussed, and the need to find the appropriate sample rate and an appropriately sized input window identified. Relevant theoretical results from dynamic systems theory are briefly introduced, and heuristics for finding the appropriate sampling rate and embedding dimension, and thence window size, are discussed. The method is applied to several time series and the resulting generalisation performance of the trained feed-forward neural network predictors is analysed. It is shown that the heuristics can provide useful information in defining the appropriate network architecture.

View PDFchevron_right

Related topics

  • Computer Science

    • Cited by

    Learning-based power prediction for geo-distributed Data Centers: weather parameter analysis
    somayyeh taheri

    Journal of Big Data, 2020

    Nowadays, in the age of big data and more data generation, there is a growing need to store and process large-scale data in real-time which has led to the deployment of cloud computing. The significant growth of the DC market has led to its rapid growth of power consumption as well as cost. By 2025, the DC market is predicted to account Abstract Nowadays, the fast rate of technological advances, such as cloud computing, has led to the rapid growth of the Data Center (DC) market as well as their power consumption. Therefore, DC power management has become increasingly important. While power forecasting can greatly help DC power management and reduce energy consumption and cost. Power forecasting predicts the potential energy failures or sudden fluctuations in power intake from utility grid. However, it is hard especially when variable renewable energies (RE) as well as free cooling such as air economizers are also used. Geo-distributed DCs face an even harder issue: since in addition to local conditions, the overall status of the entire system of collaborating DCs should also be considered. The conventional approach to forecast power consumption in such complicated cases is to construct a closed form formula for power. This is a tedious task that not only needs expert knowledge of how each single cooling or RE system works, but also needs to be done individually for each DC and repeated all over again for each new DC or change of equipment. One alternative is to use machine learning so as to learn over time how the system consumes power in varying conditions of weather, workload, and internal structure in multiple geo-distributed locations. However, due to the wide range of effective features as well as trade-off between the accuracy and processing overhead, one important issue is to obtain an optimal set of more influential features. In this study, we analyze the correlation among geo-distributed DC power patterns with their weather parameters (based on different DC situations and infrastructure) and extract a set of influential features. Afterward, we apply the obtained features to provide a power consumption forecasting model that predict the power pattern of each collaborating DC in a cloud. Our experimental results show that the proposed prediction model for geo-distributed DCs reaches the accuracy of 87.2%.

    View PDFchevron_right
    Energy time series forecasting-analytical and empirical assessment of conventional and machine learning models
    Hala Hamdoun

    2021

    Machine learning methods have been adopted in the literature as contenders to conventional methods to solve the energy time series forecasting (TSF) problems. Recently, deep learning methods have been emerged in the artificial intelligence field attaining astonishing performance in a wide range of applications. Yet, the evidence about their performance in to solve the energy TSF problems, in terms of accuracy and computational requirements, is scanty. Most of the review articles that handle the energy TSF problem are systematic reviews, however, a qualitative and quantitative study for the energy TSF problem is not yet available in the literature. The purpose of this paper is twofold, first it provides a comprehensive analytical assessment for conventional, machine learning, and deep learning methods that can be utilized to solve various energy TSF problems. Second, the paper carries out an empirical assessment for many selected methods through three real-world datasets. These datas...

    View PDFchevron_right
    A Comparative Study of Machine Learning-Based Methods for Global Horizontal Irradiance Forecasting
    Julien Eynard

    Energies, 2021

    The proliferation of photovoltaic (PV) power generation in power distribution grids induces increasing safety and service quality concerns for grid operators. The inherent variability, essentially due to meteorological conditions, of PV power generation affects the power grid reliability. In order to develop efficient monitoring and control schemes for distribution grids, reliable forecasting of the solar resource at several time horizons that are related to regulation, scheduling, dispatching, and unit commitment, is necessary. PV power generation forecasting can result from forecasting global horizontal irradiance (GHI), which is the total amount of shortwave radiation received from above by a surface horizontal to the ground. A comparative study of machine learning methods is given in this paper, with a focus on the most widely used: Gaussian process regression (GPR), support vector regression (SVR), and artificial neural networks (ANN). Two years of GHI data with a time step of ...

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved