Academia.eduAcademia.edu

Outline

Title
Abstract
Key Takeaways
Figures
Introduction
Comparison with Other Methods
Conclusions
References

A brief introduction to neural networks

Profile image of Lyle UngarLyle Ungar

1997

Abstract

Abstract Artificial neural networks are being used with increasing frequency for high dimensional problems of regression or classification. This article provides a tutorial overview of neural networks, focusing on back propagation networks as a method for approximating nonlinear multivariable functions. We explain, from a statistician's vantage point, why neural networks might be attractive and how they compare to other modern regression techniques. KEYWORDS: neural networks function approximation backpropagation.

Key takeaways
sparkles

AI

  1. Neural networks efficiently model large, complex regression and classification problems, such as handwritten ZIP code recognition.
  2. Backpropagation networks use sigmoidal activation functions and require many parameters for flexibility in approximation.
  3. Overfitting remains a challenge; cross-validation is essential for optimizing network architecture and training iterations.
  4. Neural networks lack interpretability, functioning more as black boxes compared to traditional statistical methods.
  5. The paper aims to provide a comprehensive tutorial on neural networks, focusing on their methodologies and applications.
Figures (9)
A two layer network with 9 input nodes (the 2's), a bias node and linear activation functions was started from a random initial point for the weights. The estimated weights are shown in Table 2.
A two layer network with 9 input nodes (the 2's), a bias node and linear activation functions was started from a random initial point for the weights. The estimated weights are shown in Table 2.
As a first generalization of standard linear regression, one might consider an additive adal anf the farm: Regression methods can be characterized either as projection methods, where linear or non-linear combinations of the original variables are used as predictors (such as principal components regression), or subset selection methods, where a subset of the original set. of input variables is used in the regression (e.g. stepwise regression). Table 4 characterizes several popular methods. Neural networks are a nonlinear projection method: they use non- linear combinations of optimally weighted combinations of the original variables to predict the response.
As a first generalization of standard linear regression, one might consider an additive adal anf the farm: Regression methods can be characterized either as projection methods, where linear or non-linear combinations of the original variables are used as predictors (such as principal components regression), or subset selection methods, where a subset of the original set. of input variables is used in the regression (e.g. stepwise regression). Table 4 characterizes several popular methods. Neural networks are a nonlinear projection method: they use non- linear combinations of optimally weighted combinations of the original variables to predict the response.

Related papers

neuralnet: Training of Neural Networks
F5 Gunther

2010

Artificial neural networks are applied in many situations. neuralnet is built to train multi-layer perceptrons in the context of regression analyses, i.e. to approximate functional relationships between covariates and response variables. Thus, neural networks are used as extensions of generalized linear models. neuralnet is a very flexible package. The backpropagation algorithm and three versions of resilient backpropagation are implemented and it provides a custom-choice of activation and error function. An arbitrary number of covariates and response variables as well as of hidden layers can theoretically be included. The paper gives a brief introduction to multilayer perceptrons and resilient backpropagation and demonstrates the application of neuralnet using the data set infert, which is contained in the R distribution.

View PDFchevron_right
Implementing Multiple Linear Regression model using Neural Networks
chuck Joshua P Cruz

2023

In this paper, multiple linear regression (MLR) is introduced as a valuable statistical model for predicting dependent variables based on multiple independent variables, with real-world applications ranging from precipitation estimation to bank performance analysis. However, MLR's limitations in handling complex, non-linear relationships in data are addressed by enhancing it with artificial neural networks (ANNs). ANNs excel in capturing intricate patterns, autonomously extracting features, and handling non-linear associations. The advantages of ANNs for regression tasks, such as their adaptability, scalability, and ability to generalize to unseen data, are emphasized. The paper also highlights the importance of hyperparameter tuning in optimizing the performance of MLR-ANN hybrid models and discusses the challenges and time-consuming nature of parameter experimentation, ultimately showcasing the promising potential of this combined approach for accurate predictions and adaptable learning from data.

View PDFchevron_right
Artificial neural networks: a brief literature review
Kemal Gökhan Nalbant

Journal of management and science, 2022

There are thousands of studies in the literature on the use of artificial neural network methods in estimation problems. As in all issues, modeling is one of the essential processes for the solution of estimation problems. In order to make a good prediction, it is necessary to pay attention to the necessary rules. The more attention is paid to the estimation stages, the more accurate the assessment will be, and the results can be generalized. In this study, publications in the literature on artificial neural networks were examined. This work aims to explore the studies on artificial neural networks and emphasize their importance. It is also of great importance to increase the number of studies in this field.

View PDFchevron_right
Neural networks: friends or foes?
Mark Ebden

1997

Abstract: There has been enormous interest over the past decade in the use of artificial neural networks (ANNs) for data processing applications. By and large, this interest has been well-founded. ANNs, however, offer no panacea to the data analyst. They are as prone to misuse as any other method and the details of their functioning are often clouded in mystique. This has made a firm understanding of their functioning difficult. Presents a brief introduction to the most widely applied class of ANN, the feed-forward network.

View PDFchevron_right
Neural networks. A class of flexible non-linear models for regression and classification
Manfred M Fischer
View PDFchevron_right
Treatment of multi-dimensional data to enhance neural network estimators in regression problems
Halis Altun

Expert Systems with Applications, 2007

This paper proposes and explains a data treatment technique to improve the accuracy of a neural network estimator in regression problems, where multi-dimensional input data set is highly skewed and non-normally distributed. The proposed treatment modifies the distribution characteristics of the data set. The prediction of the suspended sediment, which is an important problem in river engineering applications, will be considered as a case study. Conventional approaches lack in providing high accuracy due to the inherently employed simplicity in order to obtain empirical formulae. On the other hand, artificial neural networks are able to model the non-linear characteristics of the mechanism of the sediment transport and have a growing body of applications in diverse applications in civil engineering. It will be shown that a significant enhancement and superior score in accuracy, compared with the classical approaches, are obtainable when the proposed treatment is employed. The proposed technique is an extension to the understanding of the practical aspects of neural computing applications. Therefore the outcome of the present study is important as it is applicable to any scenario where neural network approaches are involved.

View PDFchevron_right
Chapter 5 Approximating Multivariable Functions by Feedforward Neural Nets
Paul Kainen

2013

Theoretical results on approximation of multivariable functions by feedforward neural networks are surveyed. Some proofs of universal approximation capabilities of networks with perceptrons and radial units are sketched. Major tools for estimation of rates of decrease of approximation errors with increasing model complexity are proven. Properties of best approximation are discussed. Recent results on dependence of model complexity on input dimension are presented and some cases when multivariable functions can be tractably approximated are described.

View PDFchevron_right
Neural networks
Marcello Medeiros
View PDFchevron_right
Artificial Neural Network (ANN)
Adel El-Shahat

IUPAC Standards Online, 2017

This research work presents new development in the field of natural science, where comparison is made theoretically on the efficiency of both classical regression models and that of artificial neural network models, with various transfer functions without data consideration. The results obtained based on variance estimation indicates that ANN is better which coincides with the results of Authors in the past on the efficiency of ANN over the traditional regression models. The certain conditions required for ANN efficiency over the conventional regression models were noted only that the optimal number of hidden layers and neurons needed to achieve minimum error is still open to further investigation.

View PDFchevron_right
Neural Networks: An Overview
Thomas J Hally

The purpose of this essay is to provide the reader with a general definition of Artificial Neural Networks; their functioning, applications, and a few words about what the future might allow in this area.

View PDFchevron_right

References (41)

  1. Barron, A.R. (1994), \Approximation and Estimation Bounds for Arti cial Neural Net- works." Machine Learning 14, 115-133
  2. Barron, A.R., Barron, R.L., and Wegman, E.J. (1992), \Statistical learning networks: A unifying view", in Computer Science and Statistics: Proceedings of the 20th Symposium on the Interface, edited by E.J. Wegman, 192-203.
  3. Baxt, W. G. and H. White. \Bootstrapping con dence intervals for clinical input vari- able e ects in a network trained to indentify the presence of acute myocardial infarction." Neural Computation 7 (1995) 624-638
  4. Bishop, C. M. (1995) Neural Networks for Pattern Recognition. Oxford: Clarendon Press.
  5. Broomhead, D.S., and D.Lowe (1988), \Multivarible functional interpolation and adap- tive networks." Complex Systems 2, 321-355.
  6. Buntine, W.L. and A.S. Weigend (1991), \Bayesian back-propagation." Complex Sys- tems 5, 603-643.
  7. Cheng, B. and D.M. Titterington (1994), "Neural Networks: a review from a statistical perspective, with discussion." Stat. Science 9(1) 2-54
  8. De Veaux, R. D., Psichogios, D. C., and Ungar, L. H. (1993a) \A Tale of Two Non- Parametric Estimation Schemes: MARS and Neural Networks." in Fourth International Workshop on Arti cial Intelligence and Statistics.
  9. De Veaux, R. D., Psichogios, D. C., and Ungar, L. H. (1993b) \A Comparison of Two Non-Parametric Estimation Schemes: MARS and Neural Networks." Computers and Chemical Engineering, 17(8), 819-837.
  10. De Veaux, R. D., et al. (1996) \Applying Regression Prediction Intervals to Neural Networks" in preparation.
  11. DeWeerth, S.P., Nielsen, L., Mead, C.A., Astrom, K.J. (1991), \A Simple neuron servo," IEEE Transactions on Neural Nets, 2(2), 248-251.
  12. Friedman, J.H. (1991), \Multivariate adaptive regression splines" The Annals of Statis- tics 19(1), 1-141.
  13. Friedman, J.H. and Silverman, B., (1987) \Flexible parsimonious smoothing and additive modeling ",Stanford Technical Report, Sept. 1987.
  14. Friedman, J.H., and Stuetzle W. (1981) \Projection pursuit regression," Journal of the American Statistical Association, 76, 817-823.
  15. Geman, S., and Bienenstock E. (1992), \Neural Networks and the Bias/Variance Dilemma." Neural Computation 4, 1-58.
  16. Hastie, T. and Tibshirani, R., (1986),\Generalized Additive Models." Statistical Science 1:3, 297-318.
  17. Hastie, T.J. and Tibshirani, R.J., Generalized Additive Models, Chapman and Hall, London, 1990
  18. Haykin, S., Neural Networks: A comprehensive Foundation, Macmillan, NY, 1994
  19. Hertz,J., Krogh A., and Palmer, R.G. (1991). Introduction to the Theory of Neural Computation, Addison-Welsey, Reading, MA.
  20. Holcomb, T. R., and Morari, M. (1992) \PLS/Neural Networks." Computers and Chem- ical Engineering 16:4, 393-411.
  21. Hwang, J.T. Gene and A. Adam Ding. \Prediction Intervals in Arti cial Neural Net- works" (1996).
  22. Kulkarni, A.D. (1994), \Arti cial neural networks for image understanding" Van Nos- trand Renhold, New York.
  23. Leonard J., Kramer, M., and Ungar, L.H. (1992) \Using radial basis functions to ap- proximate a function and its error bounds." IEEE Transactions on Neural Nets, 3(4), 624-627.
  24. Lippmann, R.P., (1987) \An introduction to computing with neural nets." IEEE ASSP Magazine, 4-22.
  25. MacKay, D.J.C., (1992) \A Practical Bayesian Framework for Backpropagation Net- works." Neural Computation, 4, 448-472.
  26. Mammone, R.J., editor, (1994) \Arti cial neural networks for speech and vision" Chap- man and Hall, New York.
  27. Miller, III, W.T, Sutton, R.S., and Werbos. P.J., eds. (1990) Neural Networks for Control MIT Press, Cambridge, Mass..
  28. Moody, J. and Darken, C.J., (1989) \Fast learning in networks of locally tuned processing units." Neural Computation, 1, 281-329.
  29. Narendra, K. S., and Parthasarathy, K. (1990) \Identi cation and Control of Dynamical Systems Using Neural Networks." IEEE Transactions on Neural Networks 1, 4-27.
  30. Poggio, T., and Girosi, F. (1990) \Regularization algorithms for learning that are equiv- alent to multilayer networks." Science 247, 978-982.
  31. Pomerleau, D.A. (1990) \Neural network based autonomous navigation." Vision and Navigation: The CMU Navlab Charles Thorpe, (Ed.) Kluwer Academic Publishers. Psichogios, D.C. and Ungar, L.H. (1992) \A Hybrid Neural Network | First Principles Approach to Process Modeling." AIChE Journal, 38(10), 1499-1511.
  32. Qin, S. J., and McAvoy, T. J. (1992) \Nonlinear PLS Modeling Using Neural Networks." Computers and Chemical Engineering 16:4, 379-391.
  33. Ripley, B.D. (1993) \Statistical Aspects of Neural Networks." In Networks and Chaos - Statistical and Probabilistic Aspects (eds. O.E. Barndor -Nielsen, J. L. Jensen and W.S. Kendall),40-123,Chapman and Hall, London.
  34. Ripley, B.D. (1994), \Neural networks and related methods for classi cation." Journal of the Royal Statistical Society Series B 56(3) 409-437
  35. Ripley, B.D. (1996) \Statistical Aspects of Neural Pattern Recognition and Neural Net- works" Cambridge University Press.
  36. Rumelhart, D., Hinton, G., and Williams, R. (1986) \Learning Internal Representations by Error Propagation." Parallel Distributed Processing: Explorations in the Microstruc- tures of Cognition, Vol 1: Foundations Cambridge: MIT Press, 318-362.
  37. Sanner, R.M. and Slotine J.-J.E., (1991) \Direct Adaptive Control with Gaussian Net- works." Proc. 1991 Automatic Control Conference, 3, 2153-2159.
  38. Werbos, P., (1990) \Backpropagation through time: what it does and how to do it." Proceedings of the IEEE 78:1550-60.
  39. White, H., (1989) \Learning in neural networks: a statistical perspective." Neural Computation, 1(4), 425-464.
  40. Wytho , B.J. (1993) \Backpropagation neural networks -a tutorial." Chemometrics and Intelligent laboratory systems 18(2) 115-155
  41. Zupan, J. and Gasteiger, J. (1991) \Neural networks: a new method for solving chemical problems or just a passing phase?" Analytica Chimica Acta, 248, 1-30.

Related papers

Neural Networks for Classification and Regression
Krishna Devulapalli

Biometrics & Biostatistics International Journal, 2015

Neural Networks are well known techniques for classification problems. They can also be applied to regression problems. For this, the R software packages neuralnet and RSNNS were utilized. Their application was tested with Fisher's iris dataset and a dataset from Draper and Smith and the results obtained from these models were studied.

View PDFchevron_right
ARTIFICIAL NEURAL NETWORKS AND ITS APPLICATIONS
Dr. Girish Kumar Jha

Neural networks have seen an explosion of interest over the last few years and are being successfully applied across an extraordinary range of problem domains, in areas as diverse as finance, medicine, engineering, geology, physics and biology. The excitement stems from the fact that these networks are attempts to model the capabilities of the human brain. From a statistical perspective neural networks are interesting because of their potential use in prediction and classification problems.

View PDFchevron_right
A functional approximation comparison between neural networks and polynomial regression
tai sheng huey

WSEAS Transactions on Mathematics archive, 2008

Multi-layered perceptron (MLP) neural networks are well known as universal approximators. They are often used as estimation tools in place of the classical statistical methods. The focus of this study is to compare the approximation ability of MLP with a traditional statistical regression model, namely the polynomial regression. Comparison among the single hidden layer MLP, double hidden layer MLP and polynomial regression is carried out on the basis of similar number of weights or parameters. The performance of these three categories is measured using fraction of variance unexplained (FVU). The closer the FVU value is to zero, the better the estimation result and this is associated with a higher degree of accuracy. From the empirical results obtained in this study, we conclude that overall polynomial regression performs slightly better than MLP for a similar number of parameter except for the complicated interaction function. Meanwhile, double hidden layer MLP outperforms single hidden layer MLP. The MLP is more appropriate than the polynomial regression in approximating the complicated interaction function.

View PDFchevron_right
A General Regression Neural Network
Beyblade B

This paper describes a memory-based network that provides estimates of continuous variables and converges to the underlying (linear or nonlinear) regression surface. This general regression neural network (GRNN) is a one-pass learning algorithm with a highly parallel structure. Even with sparse data in a multidimensional measurement space, the algorithm provides smooth transitions from one observed value to another. The algorithmic form can be used for any regression problem in which an assumption of linearity is not justified. The parallel network form should find use in applications such as learning the dynamics of a plant model for prediction or control.

View PDFchevron_right
The Application of Neural Networks in Prediction Problems
Rohitash Chandra

gmdh.net

This work presents the application of neural networks to real-life prediction problems. Neural networks are trained to predict three real world application problems given data for training and testing. We pre-process the actual output of the dataset by converting the values to integers and later translate them to binary strings. The sigmoidal output neurons of the feed-forward architecture predicts the output as binary values which are then translated back to integers to further compare the predicted output values of the network with the desired or actual output values from the dataset. The results for two, out of the three problems solved show that neural networks can be used to predict real-life problems similar to other inductive modeling approaches. A neural network can therefore be classified as an inductive modeling approach since it can be used for prediction, given the desired output . Finally, the performance of neural networks are then compared to GMDH.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved