Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Finding the Optimal Topology of an
Conclusions
References
All Topics
Mathematics

Finding the Optimal Topology of an Approximating Neural Network

Profile image of Stoyan CheresharovStoyan Cheresharov

Mathematics

https://doi.org/10.3390/MATH1101021
visibility

…

description

18 pages

link

1 file

Abstract

A large number of researchers spend a lot of time searching for the most efficient neural network to solve a given problem. The procedure of configuration, training, testing, and comparison for expected performance is applied to each experimental neural network. The configuration parameters—training methods, transfer functions, number of hidden layers, number of neurons, number of epochs, and tolerable error—have multiple possible values. Setting guidelines for appropriate parameter values would shorten the time required to create an efficient neural network, facilitate researchers, and provide a tool to improve the performance of automated neural network search methods. The task considered in this paper is related to the determination of upper bounds for the number of hidden layers and the number of neurons in them for approximating artificial neural networks trained with algorithms using the Jacobi matrix in the error function. The derived formulas for the upper limits of the numb...

Related papers

2012, IJARCSSE All Rights Reserved A Comprehensive Study of Artificial Neural Networks
Pooja Shukla
View PDFchevron_right
Introduction to the artificial neural networks 4y82aet9gf
زهرة الياسمين
View PDFchevron_right
Artificial Neural Networks
Vincenzo Piuri, Louis Wehenkel

Automatic Learning Techniques in Power Systems, 1998

View PDFchevron_right
Towards a Theoretical Basis For Modelling of Hidden Layer Architecture In Artificial Neural Networks
Asoka Karunananda

Artificial neural networks (ANNs) are mathematical and computational models that are inspired by the biological neural systems. Just like biological neural networks become experts by learning from the surrounding, ANNs also have the ability to be experts in the particular area by training the network. Despite of their many advantages, there are some unsolved problems in applying artificial neural networks. Determine the most efficient architecture for the given task is identified as one of those major issues. This paper provides a pruning algorithm based on the backpropagation training algorithm to obtain the optimal solution of ANN. The pruning is done according to the synaptic pruning in biological neural system. Experiments were done with some well known problems in machine learning and artificial neural networks and results show that the new model performs better than the initial network in training data sets.

View PDFchevron_right
An improved method for automatically searching near-optimal artificial Neural Networks
Leandro Almeida

2008

This paper describes an improved version of a method that automatically searches near-optimal Multilayer feedforward Artificial Neural Networks using Genetic Algorithms. This method employs an evolutionary search for simultaneous choices of initial weights, transfer functions, architectures and learning rules. Experimental results have shown that the developed method can produce compact, efficient networks with a satisfactory generalization power and with shorter training times when compared to other methods found in the literature.

View PDFchevron_right
A multi-objective memetic and hybrid methodology for optimizing the parameters and performance of artificial neural networks
Teresa Ludermir

Neurocomputing, 2010

The use of artificial neural networks implies considerable time spent choosing a set of parameters that contribute toward improving the final performance. Initial weights, the amount of hidden nodes and layers, training algorithm rates and transfer functions are normally selected through a manual process of trial-and-error that often fails to find the best possible set of neural network parameters for a specific problem. This paper proposes an automatic search methodology for the optimization of the parameters and performance of neural networks relying on use of Evolution Strategies, Particle Swarm Optimization and concepts from Genetic Algorithms corresponding to the hybrid and global search module. There is also a module that refers to local searches, including the well-known Multilayer Perceptrons, Back-propagation and the Levenberg-Marquardt training algorithms. The methodology proposed here performs the search using the aforementioned parameters in an attempt to optimize the networks and performance. Experiments were performed and the results proved the proposed method to be better than trial-and-error and other methods found in the literature.

View PDFchevron_right
Mathematical Methods for the Implementation of Neural Networks.
Eric Mjolsness

1996

Abstract: We present a novel optimizing network architecture with applications in vision, learning, pattern recognition and combinatorial optimization. This architecture is constructed by combining the following techniques:(1) deterministic annealing,(2) self-amplification,(3) algebraic transformations,(4) clocked objectives, and (5) soft assign. Deterministic annealing in conjunction with self-amplification avoids poor local minima and ensures that a vertex of the hypercube is reached.

View PDFchevron_right
A procedure for determining the topology of multilayer feedforward neural networks
Julian Morris, Ming Tham

Neural Networks, 1994

Thts contrtbutton describes a procedure for determmmg the opttmal topology of a statw three-layer neural network The method ts based on a canomcal decomposttion techmque that estabhshes a hnk between the number of neurons m each htdden layer and the dtmensions of the subspaces of the canomcal decomposmon. It ensures that the structure ts suffictently complex to characterise the process being modelled by the network, and also the untcity of the approxtmatton A practtcal procedure to tmplement this method is then outhned and applied to both stmulated and real process data Keywords--Neural network topology, Canonical form, Number of neurons.

View PDFchevron_right
Training Optimization for Artificial Neural Networks
Primitivo Luna, Ralejoll Lama, Rosa Rosas

2011

C I E N C I A e r g o s u m , V o l . 1 7 -3 , n o v i e m b r e 2 0 1 0 -f e b r e r o 2 0 1 1 . Universidad Autónoma del Estado de México, Toluca, México. Pp. 313-317. Optimizaciòn del entrenamiento para Redes Neuronales Artificiales Resumen. Debido a la habilidad para modelar problemas complejos, actualmente las Redes Neuronales Artificiales (nn) son muy populares en Reconocimiento de Patrones, Minería de Datos y Aprendizaje Automático. No obstante, el elevado costo computacional asociado a la fase en entrenamiento, cuando grandes bases de datos son utilizados, es su principal desventaja. Con la intención de disminuir el costo computacional e incrementar la convergencia de la nn, el presente trabajo analiza la conveniencia de realizar pre-procesamiento a los conjuntos de datos. De forma específica, se evalúan los métodos de grafo de vecindad relativa (rng), grafo de Gabriel (gg) y el método basado en los vecinos envolventes k-ncn. Los resultados experimentales muestran la factibilidad y las múltiples ventajas de esas metodologías para solventar los problemas descritos previamente. Palabras clave: redes neuronales artificiales, perceptrón multicapa, redes de función de base radial, máquinas de vectores soporte, preprocesado de datos. Abstract. Nowadays, with the capacity to model complex problems, the artificial Neural Networks (nn) are very popular in the areas of Pattern Recognition, Data Mining and Machine Learning. Nevertheless, the high computational cost of the learning phase when big data bases are used is their main disadvantage. This work analyzes the advantages of using pre-processing in data sets In order to diminish the computer cost and improve the nn convergence. Specifically the Relative Neighbor Graph (rng), Gabriel's Graph (gg) and k-ncn methods were evaluated. The experimental results prove the feasibility and the multiple advantages of these methodologies to solve the described problems.

View PDFchevron_right
EAI732: Intelligent Systems Assignment 2: Neural Networks
wihan Booyse
View PDFchevron_right

References (45)

  1. Rozycki, P.; Kolbusz, J.; Wilamowski, B.M. Dedicated deep neural network architectures and methods for their training. In Proceedings of the IEEE 19th International Conference on Intelligent Engineering Systems (INES), Bratislava, Slovakia, 3-5 September 2015.
  2. Torres, J.J.; Munoz, M.A.; Marro, J.; Garrido, P.L. Influence of topology on the performance of a neural network. Neurocomputing 2004, 58-60, 229-234.
  3. Emmert-Streib, F. Influence of the neural network topology on the learning dynamics. Neurocomputing 2006, 69, 1179-1182.
  4. Lv, C.; Xing, Y.; Zhang, J.; Na, X.; Li, Y.; Liu, T.; Cao, D.; Wang, F.-Y. Levenberg-Marquardt Backpropagation Training of Mul- tilayer Neural Networks for State Estimation of a Safety-Critical Cyber-Physical System. IEEE Trans. Ind. Inform. 2018, 14, 3436- 3446.
  5. Sapna, S.; Tamilarasi, A.; Kumar, M.; Backpropagation learning algorithm based on Levenberg Marquardt algorithm. Comput. Sci. Inf. Technol. 2012, 7, 393-398.
  6. Botev, A.; Ritter, H.; Barber, D. Practical Gauss-Newton Optimisation for Deep Learning. In Proceedings of the 34-th Interna- tional Conference on Machine Learning PMLR 70, Sydney, Australia, 6-11 August 2017; pp. 557-565.
  7. Gratton, S.; Lawless, A.; Nichols, N. Approximate Gauss-Newton Methods for Nonlinear Least Squares Problems. SIAM J. Optim. 2017, 18, 106-132.
  8. Okut, H. Bayesian Regularized Neural Networks for Small n Big p Data. In Artificial Neural Networks-Models and Applications; IntechOpen: London, UK, 2016.
  9. Gouravaraju, S.; Narayan, J.; Sauer, R.; Gautam, S. A Bayesian regularization-backpropagation neural network model for peel- ing computations, J. Adhes. 2023, 99, 92-115.
  10. Chel, H.; Majumder, A.; Nandi, D. Scaled Conjugate Gradient Algorithm in Neural Network Based Approach for Handwritten Text Recognition. Commun. Comput. Inf. Sci. 2011, 204, 196-210.
  11. Babani, L.; Jadhav, S.; Chaudhari, B. Scaled Conjugate Gradient Based Adaptive ANN Control for SVM-DTC Induction Motor Drive. In Artificial Intelligence Applications and Innovations. AIAI 2016; Iliadis, L., Maglogiannis, I., Eds; Springer: Cham, Switzer- land, 2016; Volume 475, pp. 384-395.
  12. Goldfarb, D.; Ren, Y.; Bahamou, A. Practical Quasi-Newton Methods for Training Deep Neural Networks. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, 6-12 December 2020.
  13. Wills, A.; Schön, T.; Jidling, C. A fast quasi-Newton-type method for large-scale stochastic optimization. IFAC-Papers 2020, 53, 1249-1254.
  14. Hunter, D.; Yu, H.; Pukish III, M.S.; Kolbusz, J.; Wilamowski, B.M. Selection of proper neural network sizes and architectures: A comparative study. IEEE Trans. Ind. Inform. 2012, 8, 228-240.
  15. Kuri, A. The Best Neural Network Architecture. In Proceedings of the Mexican International Congress on Artificila Intelligence, Monterrey, NL, Mexico, 24-29 October 2014.
  16. Khalil, K.; Eldash, O.; Kumar, A.; Bayoumi, M. An Efficient Approach for Neural Network Architecture. In proceedings of the 25th IEEE International Conference on Electronics, Circuits and Systems (ICECS), Bordeaux, France, 9-12 December 2018.
  17. Jinchuan, K.; Xinzhe, L. Empirical analysis of optimal hidden neurons in neural network modeling for stock prediction. In Proceedings of the Pacific-Asia Workshop on Computational Intelligence and Industrial Application, Wuhan, China, 19-20 December 2008; Volume 2, pp. 828-832.
  18. Alvarez, J.M.; Salzmann, M. Learning the Number of Neurons in Deep Networks. Adv. Neural Inf. Process. Syst. 2016, 29, 1-9.
  19. Li, J.Y.; Chow, T.W.S.; Yu, Y.L. Estimation theory and optimization algorithm for the number of hidden units in the higher- order feedforward neural network. In Proceedings of the IEEE International Conference on Neural Networks, Perth, WA, Australia, 27 November-1 December 1995; pp. 1229-1233.
  20. Tamura, S.; Tateishi, M. Capabilities of a four-layered feedforward neural network: Four layers versus three, In Proceedings of the IEEE Transactions on Neural Networks, March 1997, 8, 251-255.
  21. Sheela, K.G.; Deepa, S.N. Review on Methods to Fix Number of Hidden Neurons in Neural Networks. Math. Probl. Eng. 2013, 2013, 425740.
  22. Madhiarasan, M.; Deepa, S.N. Comparative analysis on hidden neurons estimation in multi layer perceptron neural networks for wind speed forecasting. Artif. Intell. Rev. 2016, 48, 449-471.
  23. Xu, S.; Chen, L. A novel approach for determining the optimal number of hidden layer neurons for FNN's and its application in data mining. In Proceedings of the 5th International Conference on Information Technology and Applications, 23-26 July 2008; pp. 683-686.
  24. Shibata, K.; Ikeda, Y. Effect of number of hidden neurons on learning in large-scale layered neural networks. In Proceedings of the ICROS-SICE International Joint Conference, Fukuoka International Congress Center, Japan, Cairns, Queensland, Australia, 18-21 August 2009; pp. 5008-5013.
  25. Yotov, K.; Hadzhikolev, E.; Hazdhikoleva, S. Determining the Number of Neurons in Artificial Neural Networks for Approximation, Trained with Algorithms Using the Jacobi Matrix. TEM J. 2020, 9, 1320-1329.
  26. Shen, H.-Y.; Wang, Z.; Gao, C.-Y. Determining the number of BP neural network hidden layer units. J. Tianjin Univ. Technol. 2008, 24, 13-15.
  27. Ibnu Choldun, R.M.; Santoso, J.; Surendro, K. Determining the number of hidden layers in neural network by using principal component analysis. Adv. Intell. Syst. Comput. 2020, 1038, 490-500.
  28. Stathakis, D. How many hidden layers and nodes? Int. J. Remote Sens. 2009, 30, 2133-2147.
  29. Hanay, Y.S.; Arakawa, S.; Murata, M. Network topology selection with multistate neural memories. Expert Syst. Appl. 2015, 42, 3219-3226.
  30. Perzina, R. Self-learning Genetic Algorithm for Neural Network Topology Optimization. Smart Innov. Syst. Technol. 2015, 38, 179-188. 10.1007/978-3-319-19728-9_15.
  31. Vizitiu, I.-C.; Popescu, F. GANN system to optimize both topology and neural weights of a feedforward neural network. In Proceedings of the 8th International Conference on Communications, Sintok, Kedah, Malaysia, 1-3 October, 2022.
  32. White, D.; Ligomenides, P. GANNet: A genetic algorithm for optimizing topology and weights in neural network design. Lect. Notes Comput. Sci. 1995, 686, 322-327.
  33. Arena, P.; Caponetto, R.; Fortuna, L.; Xibilia, M.G. (n.d.). Genetic algorithms to select optimal neural network topology. In Proceedings of the 35th Midwest Symposium on Circuits and Systems, Washington, DC, USA, 9-12 August 1992.
  34. Leon, F. Optimizing neural network topology using Shapley value. In Proceedings of the 18th International Conference on System Theory, Control and Computing (ICSTCC), Sinaia, Romania, 17-19 October 2014.
  35. Kuroda, K.; Hasegawa, M. Method for Estimating Neural Network Topology Based on SPIKE-Distance. Lect. Notes Comput. Sci. 2016, 9886, 91-98.
  36. Curteanu, S.; Leon, F.; Furtuna, R.; Dragoi, E.N.; Curteanu, N. Comparison between different methods for developing neural network topology applied to a complex polymerization process. In Proceedings of the 2010 International Joint Conference on Neural Networks (IJCNN). Barcelona, Spain, 18-23 July 2010.
  37. Guliyev, N.J.; Ismailov, V.E. On the approximation by single hidden layer feedforward neural networks with fixed weights. Neural Netw. 2018, 98, 296-304.
  38. Thomas, A.J.; Petridis, M.; Walters, S.D.; Gheytassi, S.M.; Morgan, R.E. Two Hidden Layers are Usually Better than One. Com- mun. Comput. Inf. Sci. 2017, 744, 279-290.
  39. Nakama, T. Comparisons of Single-and Multiple-Hidden-Layer Neural Networks. Lect. Notes Comput. Sci. 2011, 6675, 270-279.
  40. Ibnu, C.R.M.; Santoso, J.; Surendro, K. Determining the Neural Network Topology. In Proceedings of the 8th International Conference on Software and Computer Applications-ICSCA. 2019, 19, 357-362.
  41. MatLab. Available online: https://www.mathworks.com/products/matlab.html (accessed on 8 October 2022).
  42. Börlin, N. Nonlinear Optimization Least Squares Problems-The Gauss-Newton method. Available online: https://www8.cs.umu.se/kurser/5DA001/HT07/lectures/lsq-handouts (accessed on 30 December 2022).
  43. Cartis, C. Mathematical Institute, University of Oxford. Linear and nonlinear least-squares problems; the Gauss-Newton method. Available online: https://courses-archive.maths.ox.ac.uk/node/view_material/4898. (accessed on 8 October 2022).
  44. Madsen, K.; Nielsen, H.B.; Tingleff, O. Methods for Non-Linear Least Squares Problems, 2004. Available online: http://www2.imm.dtu.dk/pubdb/edoc/imm3215.pdf. (accessed on 8 October 2022).
  45. Disclaimer/Publisher's Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Related papers

Optimization of the Neural Networks Parameters
Yousef Abuzir

2017

Neural Networks (NN) is an effective approach used in many fields such as medicine, industry, security, stock market prediction, character recognition, image processing and many other fields. The main problem in the use of Artificial Neural Networks (ANN) is the control of parameters, since there is no explicit and specific way to determine the optimal values of Neural Network parameters. The aim of this study is to choose the best parameters that can be used to model and optimize the number of hidden layers, number of neurons contained in there, momentum, delta rule, transition functions and multidimensional network.

View PDFchevron_right
A Novel Technique for Optimizing the Hidden Layer Architecture in Artificial Neural Networks
Asoka Karunananda

Artificial neural networks have been showed their effectiveness in many real world problems such as signal processing, pattern recognition, and classification problems. Although they provide highly generalized solutions, we find several unanswered problems in using artificial neural networks. Determining the most appropriate architecture of artificial neural network is identified as one of those major problems. Generally, the performance of a neural network strongly depends on the size of the network. By increasing the number of layers generalization ability can be improved. However, this solution may not be computationally optimized. On the other hand, too many hidden neurons may over-train the data and which cause the poor generalization. Also, too few neurons under-fit the data and hence, network may not train the data properly. Thus, both too many and too few neurons show bad generalization. Therefore, determining the most suitable architecture is very important in artificial neural networks. As such, a large number of researchers have been carried out to model the hidden layer architecture by using various techniques. These techniques can be categorized as pruning techniques and constructive techniques. Pruning algorithms start with an oversized network and remove nodes until the optimal architecture occurs [1],[2],[3],[4] and [12]. Constructive algorithms [5],[6],[7],[8] do the other way. They build the appropriate neural network during the training process by adding hidden layers, nodes and connection weights to a minimal architecture. However, most of these methods are confined to networks with small number of neurons or single hidden layer neural networks. Hence, they have not addressed the existing problem of hidden layer architecture properly. In this paper, a new pruning algorithm based on backpropagation training [11] has been proposed to design the optimal neural network. The optimal solution is obtained by two steps. First, the number of hidden layers in the most efficient network is determined. Then the network tends to the optimal solution by removing all unimportant nodes from each layer. The removable nodes are identified through the delta values of hidden neurons [9],[10]. The choosing of delta values was based on the fact that the delta values of the hidden layers are used to compute the error term of the next training cycle. Hence, delta value is a significant factor in error term. Thus, the delta values are used to identify the less saliency neurons and remove them from hidden neurons so that the error term tends to the desired limit faster than the backpropagation training. The approaches of the other researchers are discussed in the next section. Section III describes the new algorithm and how to use the delta values in optimization of hidden layer architecture. The experimental method and results are discussed in section IV. Finally, section V presents the conclusions.

View PDFchevron_right
Comparative performance of some popular artificial neural network algorithms on benchmark and function approximation problems
Bp Dubey

Pramana-journal of Physics, 2010

We report an inter-comparison of some popular algorithms within the artificial neural network domain (viz., Local search algorithms, global search algorithms, higher order algorithms and the hybrid algorithms) by applying them to the standard benchmarking problems like the IRIS data, XOR/N-Bit parity and Two Spiral. Apart from giving a brief description of these algorithms, the results obtained for the above

View PDFchevron_right
Optimization of multi-layer artificial neural networks using delta values of hidden layers
Asoka Karunananda

2013 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), 2013

The number of hidden layers is crucial in multilayer artificial neural networks. In general, generalization power of the solution can be improved by increasing the number of layers. This paper presents a new method to determine the optimal architecture by using a pruning technique. The unimportant neurons are identified by using the delta values of hidden layers. The modified network contains fewer numbers of neurons in network and shows better generalization. Moreover, it has improved the speed relative to the back propagation training. The experiments have been done with number of test problems to verify the effectiveness of new approach.

View PDFchevron_right
An Optimization Methodology for Neural Network Weights and Architectures
Cleber Zanchettin

IEEE Transactions on Neural Networks, 2006

This paper introduces a methodology for neural net- work global optimization. The aim is the simultaneous optimiza- tion of multilayer perceptron (MLP) network weights and archi- tectures, in order to generate topologies with few connections and high classification performance for any data sets. The approach combines the advantages of simulated annealing, tabu search and the backpropagation training algorithm in order

View PDFchevron_right

Related topics

  • Mathematics
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved