Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Methodology
Computer Simulations and Results
Conclusion
References

Differential Neural Networks (DNN)

Profile image of Juan Gabriel Avina-CervantesJuan Gabriel Avina-Cervantes

IEEE Access

https://doi.org/10.1109/ACCESS.2020.3019307
visibility

…

description

9 pages

link

1 file

Abstract

In this work, we propose an artificial neural network topology to estimate the derivative of a function. This topology is called a differential neural network because it allows the estimation of the derivative of any of the network outputs with respect to any of its inputs. The main advantage of a differential neural network is that it uses some of the weights of a multilayer neural network. Therefore, a differential neural network does not need to be trained. First, a multilayer neural network is trained to find the best set of weights that minimize an error function. Second, the weights of the trained network and its neuron activations are used to build a differential neural network. Consequently, a multilayer artificial neural can produce a specific output, and simultaneously, estimate the derivative of any of its outputs with respect to any of its inputs. Several computer simulations were carried out to validate the performance of the proposed method. The computer simulation results showed that differential neural networks are capable of estimating with good accuracy the derivative of a function. The method was developed for an artificial neural network with two layers; however, the method can be extended to more than two layers. Similarly, the analysis in this study is presented for two common activation functions. Nonetheless, other activation functions can be used as long as the derivative of the activation function can be computed. INDEX TERMS Differential neural network, artificial intelligence, neural network structure, derivative estimation, multilayer network.

Figures (11)
FIGURE 1. Structure of the artificial neural network.
FIGURE 1. Structure of the artificial neural network.
B. INTERNAL DERIVATIVE In this section, the internal derivative dq;/dx; indicated in Figure 2 is computed. From this figure, it can be seen that the values of g; can be computed as
B. INTERNAL DERIVATIVE In this section, the internal derivative dq;/dx; indicated in Figure 2 is computed. From this figure, it can be seen that the values of g; can be computed as
FIGURE 4. Algorithm to estimate the derivative. FIGURE 3. Differential neural network (DNN).
FIGURE 4. Algorithm to estimate the derivative. FIGURE 3. Differential neural network (DNN).
FIGURE 5. Derivative estimation for sin(x).
FIGURE 5. Derivative estimation for sin(x).
FIGURE 6. Error for the approximation of the derivative of sin(x).
FIGURE 6. Error for the approximation of the derivative of sin(x).
FIGURE 9. Derivative estimation for /n(x). FIGURE 8. Error for the approximation of the derivative of x?.
FIGURE 9. Derivative estimation for /n(x). FIGURE 8. Error for the approximation of the derivative of x?.
TABLE 2. Mean-squared error of the derivative estimation.
TABLE 2. Mean-squared error of the derivative estimation.
FIGURE 10. Error for the approximation of the derivative of /n(x).
FIGURE 10. Error for the approximation of the derivative of /n(x).

Related papers

An Application of Neural Networks to Solve Ordinary Differential Equations
somayeh ezadi

2013

In this paper, we introduce a hybrid approach based on modied neural networks and optimization teqnique to solve ordinary dierential equation. Using modied neural net- work makes that training points should be selected over the open interval (a;b) without training the network in the range of rst and end points. Therefore, the calculating volume involving computational error is reduced. In fact, the training points depending on the dis- tance (a;b) selected for training neural networks are converted to similar points in the open interval (a;b) by using a new approach, then the network is trained in these similar areas. In comparison with existing similar neural networks proposed model provides solutions with high accuracy. Numerical examples with simulation results illustrate the eectiveness of the proposed model.

View PDFchevron_right
Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks
Maxwell Stinchcombe

Neural Networks, 1990

We give conditions ensuring that multilayer jeedJorward networks with as Jew as a single hidden layer and an appropriately smooth hidden layer activation fimction are capable of arbitrarily accurate approximation to an arbitrao' function and its derivatives. In fact, these networks can approximate functions that are not dtifferentiable in the classical sense, but possess only a generalized derivative, as is the case jor certain piecewise dtlf[erentiable Junctions. The conditions imposed on the hidden layer activation function are relatively mild: the conditions imposed on the domain of the fimction to be approximated have practical intplications. Our approximation results provide a previously missing theoretical justification jor the use of multilayer /~'ed/brward networks in applications requiring simultaneous approximation of a function and its derivatives.

View PDFchevron_right
Approximating a function and its derivatives using MSE-optimal linear combinations of trained feedforward neural networks
Sherif Hashem

1993

In this paper, we show that using MSE-optimal linear combinations of a set of trained feedforward networks may signi cantly improve the accuracy of approximating a function and its rst and second order derivatives. Our results are compared to the accuracies achieved by the single best network and by the simple averaging of the outputs of the trained networks.

View PDFchevron_right
The numerical solution of linear ordinary differential equations by feedforward neural networks
Ariel Fernandez

Mathematical and Computer Modelling, 1994

is demonstrated, through theory and examples, how it is possible to construct directly and noniteratively a feedforward neural network 'co approximate arbitrary linear ordinary differential equations. The method, using the hard limit transfer function, is linear in storage and processing time, and the 152 norm of the network approximation error decreases quadratically with the increasing number of hidden layer neurons. The construction requires imposing certain constraints on the values of the input, bias, and output weights, and the attribution of certain roles to each of these parameters. All results presented used the hard limit transfer function. However, the noniterative approach should also be applicable to the use of hyperbolic tangents, sigmoids, and radial basis functions.

View PDFchevron_right
Algorithm for building a neural network for function approximation
Sarvesh Chandra

IEE Proceedings - Circuits, Devices and Systems, 2002

A technique for constructing a multilayer tree-structured neural network, which provides a continuous, piece-wise linear function approximation, is presented. The method uses a growth network with linear threshold neurons. Neurons are added to a binary tree until the approximation error at all sampling points is brought down to within a specified+d. The number of neurons in the constructed network depends on the samples provided as well as the specified tolerance A , thus enabling a trade-off between accuracy and network size. In comparison to approaches such as backpropagation, the proposed technique requires no assumptions regarding the number of neurons, learning rate, momentum term, or initial weight values. It also does not suffer from problems of local minima. Examples are presented to illustrate the effectiveness of the technique.

View PDFchevron_right
Application of Relative Derivation Terms by Polynomial Neural Networks
Ladislav Zjavka

2013

A lot of problems involve unknown data relations, which can define a derivative based model of dependent variables generalization. Standard soft- computing methods (as artificial neural networks or fuzzy rules) apply usual absolute interval values of input variables. The new proposed differential polynomial neural network makes use of relative data, which can better describe the character regarding a wider range of input values. It constructs and resolves an unknown partial differential equation, using fractional polynomial sum derivative terms of relative data changes. This method might be applied to solve problems concerned a visual pattern generalization or complex system modeling.

View PDFchevron_right
Introducing the bounded derivative network—superceding the application of neural networks in control
John Guiver

Journal of Process Control, 2005

The Bounded Derivative Network (BDN), the analytical integral of a neural network, is a natural and elegant evolution of universal approximating technology for use in automatic control schemes. This modeling approach circumvents the many real problems associated with standard neural networks in control such as model saturation (zero gain), arbitrary model gain inversion, Ôblack boxÕ representation and inability to interpolate sensibly in regions of sparse excitation. Although extrapolation is typically not an advantage unless the understanding of the process is complete, the BDN can incorporate process knowledge in order that its extrapolation capability is inherently sensible in areas of data sparsity. This ability to impart process knowledge on the BDN model enables it to be safely incorporated into a model based control scheme.

View PDFchevron_right
On learning the derivatives of an unknown mapping with multilayer feedforward networks
A.ronald Gallant

Neural Networks, 1992

Recently, multiple input, single output, single hidden layer, feedforward neural networks have been shown to be capable of approximating a nonlinear map and its partial derivatives. Specifically, neural nets have been shown to be dense in various Sobolev spaces (Hornik, Stinchcombe and White, 1989). Building upon this result, we show that a net can be trained so that the map and its derivatives are learned. Specifically, we use a result of Gallant (1987b) to show that least squares and similar estimates are strongly consistent in Sobolev norm provided the number of hidden units and the size of the training set increase together. We illustrate these results by an applic~tion to the inverse problem of chaotic dynamics: recovery of a nonlinear map from a time series of iterates. These results extend automatically to nets that embed the single hidden layer, feedforward network as a special case.

View PDFchevron_right
Author's personal copy Differentiating adaptive Neuro-Fuzzy Inference System for accurate function derivative approximation
omid khayat

Function and its partial derivative approximation based upon a set of discrete dataset are important issues in soft computing. Several function approximators have been presented most of them fits a model to the dataset so that the Mean Squared Error is minimized. In this paper, we propose to calculate the derivative of the Neuro-Fuzzy function approximator directly according to the parametric structure of the system and the available dataset. A criterion for derivative approximation is defined based on a combination of MSE and Approximate Entropy. According to this criterion, the superiority of the Neuro-Fuzzy model is demonstrated in comparison with some other types of Artificial Neural Networks and Polynomial models.

View PDFchevron_right
Solving Differential Equations Using Feedforward Neural Networks
Isaac Santos

Computational Science and Its Applications – ICCSA 2021

In this work we explore the use of deep learning models based on deep feedforward neural networks to solve ordinary and partial differential equations. The illustration of this methodology is given by solving a variety of initial and boundary value problems. The numerical results, obtained based on different feedforward neural networks structures, activation functions and minimization methods, were compared to each other and to the exact solutions. The neural network was implemented using the Python language, with the Tensorflow library.

View PDFchevron_right

References (34)

  1. S. Marsland, Machine Learning: An Algorithmic Perspective (Chapman & Hall/CRC Machine Learning & Pattern Recognition), 2nd ed. Boca Raton, FL, USA: CRC Press, 2015, pp. 190-210.
  2. M. I. Jordan and T. M. Mitchell, ''Machine learning: Trends, perspectives, and prospects,'' Science, vol. 349, no. 6245, pp. 255-260, Jul. 2015.
  3. C. Yin, Y. Zhu, J. Fei, and X. He, ''A deep learning approach for intru- sion detection using recurrent neural networks,'' IEEE Access, vol. 5, pp. 21954-21961, 2017.
  4. W. Liu, M. Zhang, Z. Luo, and Y. Cai, ''An ensemble deep learning method for vehicle type classification on visual traffic surveillance sensors,'' IEEE Access, vol. 5, pp. 24417-24425, 2017.
  5. B. Jin, Z. Jing, and H. Zhao, ''Incremental and decremental extreme learning machine based on generalized inverse,'' IEEE Access, vol. 5, pp. 20852-20865, 2017.
  6. S. Ledesma, M.-A. Ibarra-Manzano, E. Cabal-Yepez,
  7. D.-L. Almanza-Ojeda, and J.-G. Avina-Cervantes, ''Analysis of data sets with learning conflicts for machine learning,'' IEEE Access, vol. 6, pp. 45062-45070, 2018.
  8. R. J. Martin, S. L. Swapna, and S. Sujatha, ''Adopting machine learning models for data analytics-A technical note,'' Int. J. Comput. Sci. Eng., vol. 6, no. 10, pp. 359-364, Oct. 2018.
  9. Y. LeCun, Y. Bengio, and G. Hinton, ''Deep learning,'' Nature, vol. 521, pp. 436-444, May 2015.
  10. V. Mnih, K. Kavukcuoglu, and D. Silver, ''Human-level control through deep reinforcement learning,'' Nature, vol. 518, pp. 529-533, 2015.
  11. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, ''A survey of deep neural network architectures and their applications,'' Neurocomput- ing, vol. 234, pp. 11-26, Apr. 2017.
  12. K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, ''Deep convolutional neural network for inverse problems in imaging,'' IEEE Trans. Image Process., vol. 26, no. 9, pp. 4509-4522, Sep. 2017.
  13. M. Umar, F. Amin, H. A. Wahab, and D. Baleanu, ''Unsupervised con- strained neural network modeling of boundary value corneal model for eye surgery,'' Appl. Soft Comput., vol. 85, Dec. 2019, Art. no. 105826.
  14. M. A. Z. Raja, J. A. Khan, and I. M. Qureshi, ''A new stochastic approach for solution of Riccati differential equation of fractional order,'' Ann. Math. Artif. Intell., vol. 60, nos. 3-4, pp. 229-250, Dec. 2010.
  15. S. Mall and S. Chakraverty, ''Application of Legendre neural network for solving ordinary differential equations,'' Appl. Soft Comput., vol. 43, pp. 347-356, Jun. 2016.
  16. M. A. Z. Raja, M. A. Manzar, and R. Samar, ''An efficient computational intelligence approach for solving fractional order Riccati equations using ANN and SQP,'' Appl. Math. Model., vol. 39, nos. 10-11, pp. 3075-3093, Jun. 2015.
  17. A. Jafarian, S. Measoomy Nia, A. K. Golmankhaneh, and D. Baleanu, ''On artificial neural networks approach with new cost functions,'' Appl. Math. Comput., vol. 339, pp. 546-555, Dec. 2018.
  18. A. J. Meade and A. A. Fernandez, ''The numerical solution of linear ordinary differential equations by feedforward neural networks,'' Math. Comput. Model., vol. 19, no. 12, pp. 1-25, Jun. 1994.
  19. I. E. Lagaris, A. Likas, and D. I. Fotiadis, ''Artificial neural networks for solving ordinary and partial differential equations,'' IEEE Trans. Neural Netw., vol. 9, no. 5, pp. 987-1000, Sep. 1998.
  20. S. Mall and S. Chakraverty, ''Comparison of artificial neural network architecture in solving ordinary differential equations,'' Adv. Artif. Neural Syst., vol. 2013, pp. 1-12, Jan. 2013.
  21. M. Pakdaman, A. Ahmadian, S. Effati, S. Salahshour, and D. Baleanu, ''Solving differential equations of fractional order using an optimization technique based on training artificial neural network,'' Appl. Math. Com- put., vol. 293, pp. 81-95, Jan. 2017.
  22. F. Rostami and A. Jafarian, ''A new artificial neural network structure for solving high-order linear fractional differential equations,'' Int. J. Comput. Math., vol. 95, no. 3, pp. 528-539, Mar. 2018.
  23. S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed. London, U.K.: Pearson, 2020, pp. 122-204.
  24. T. Masters, Deep Belief Nets in C++ and CUDA C: Restricted Boltzmann Machines and Supervised Feedforward Networks, vol. 1. Ithaca, NY, USA: Apress, 2017, pp. 11-25.
  25. T. Masters, Deep Belief Nets in C++ and CUDA C: Convolutional Nets, vol. 3. Ithaca, NY, USA: Apress, 2018, pp. 28-37.
  26. J. D. Kelleher, B. M. Namee, and A. D'Arcy, Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies, 2nd ed. Cambridge, MA, USA: MIT Press, 2020.
  27. C. C. Aggarwal, Neural Networks and Deep Learning: A Textbook, 1st ed. Cham, Switzerland: Springer, 2018, pp. 53-103.
  28. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning (Adaptive Computation and Machine Learning Series). Cambridge, MA, USA: MIT Press, 2016, pp. 267-309.
  29. S. Ledesma, M.-A. Ibarra-Manzano, M.-G. Garcia-Hernandez, and D.-L. Almanza-Ojeda, ''Neural lab a simulator for artificial neural net- works,'' in Proc. IEEE Comput. Conf., Jul. 2017, pp. 716-721.
  30. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numer- ical Recipes in C++: The Art of Scientific Computing, 3rd ed. Cambridge, U.K.: Cambridge Univ. Press, 2007, pp. 549-555.
  31. S. Ledesma, M. Torres, D. Hernandez, G. Avina, and G. Garcia, ''Tem- perature cycling on simulated annealing for neural network learning,'' in Proc. MICAI. Berlin, Germany: Springer, 2007, pp. 161-171.
  32. S. Ledesma, G. Avina, and R. Sanchez, Practical Considerations for Sim- ulated Annealing Implementation (Simulated Annealing Book). London, U.K.: IntechOpen, 2008, pp. 401-420.
  33. K. L. Du and M. N. S. Swamy, Neural Networks and Statistical Learning, 2nd ed. London, U.K.: Springer, 2019, pp. 143-164.
  34. SERGIO LEDESMA received the M.S. degree from the University of Guanajuato, Mexico, while working on the setup of Internet, and the Ph.D. degree from the Stevens Institute of Technol- ogy, Hoboken, NJ, in 2001. After graduating, he worked for Barclays Bank as part of the IT- HR Group. He has worked as a Software Engineer for several years, and he is also the Creator of the Software Neural Laboratory, Wintempla, and TexLab. He is currently a Research Professor with the University of Guanajuato. He is on a sabbatical stay with the University of Ottawa, Canada. His research interests include artificial intelligence and software engineering. DORA-LUZ ALMANZA-OJEDA received the B.S. degree in electronics engineering and the M.S. degree in electrical engineering from the University of Guanajuato, Salamanca, Mexico, in 2003 and 2005, respectively, and the Ph.D. degree from Paul Sabatier University, Toulouse, France, in 2011. She is currently an Assistant Pro- fessor with the Electronics Engineering Depart- ment, University of Guanajuato. Her research interests include embedded vision for controlling autonomous robots and real time systems. MARIO-ALBERTO IBARRA-MANZANO (Member, IEEE) received the B.Eng. degree in communication and electronic engineering and the M.Eng. degree in electric from the University of Guanajuato, Salamanca, Mexico, in 2003 and 2006, respectively, and the Ph.D. degree (Hons.) from the Institut National des Sciences Appliquées, Toulouse, France, in 2011. He is currently an Assistant Professor with the Electronics Engineering Department, Universidad de Guanajuato. His research interests include digital design on FPGA for image processing applied on autonomous robots and real time systems.

Related papers

Differential Learning Algorithm for Artificial Neural Networks
Narendranath Udupa

International Journal of Computer Applications, 2010

Artificial Neural Networks (ANN) are extremely useful to relate the nonlinearly depending outputs with the inputs. Various architectures are available for the ANNs to speedup the training period and reduce the square error. In this paper, new classes of neural networks with differential feedback are presented. The different orders of differential feed back form a manifold of hyperplanes. Interesting properties of this differentially fed ANN (DANN) are derived through these hyperplanes.

View PDFchevron_right
Construction and adjustment of differential polynomial neural network
Ladislav Zjavka

Journal of Engineering and Computer Innovations, 2011

Artificial neural networks in general are used to identify patterns according to their entire relationship, responding to related patterns with a similar output of applying absolute values of variables. However, a lot of real data contain some unknown relations of variables. Learning of these dependencies could be a new way of modelling complex systems instead of usual time series prediction based on pattern similarity. Differential polynomial neural network, which constructs a differential equation of fractional terms using multi-parametric polynomial functions, is a new type of neural network developed by the author. Its functionality is based on principles, which are applied in human brain learning. The brain does not utilize absolute values of variables, but relative ones, which are created by time-delayed dynamic periodic activation functions of biological neurons. They take part in differential equation composition as partial derivative terms, describing a relative change of particular dependent variables.

View PDFchevron_right
Comparison of Artificial Neural Network Architecture in Solving Ordinary Differential Equations
S. Chakraverty

Advances in Artificial Neural Systems, 2013

This paper investigates the solution of Ordinary Differential Equations (ODEs) with initial conditions using Regression Based Algorithm (RBA) and compares the results with arbitrary- and regression-based initial weights for different numbers of nodes in hidden layer. Here, we have used feed forward neural network and error back propagation method for minimizing the error function and for the modification of the parameters (weights and biases). Initial weights are taken as combination of random as well as by the proposed regression based model. We present the method for solving a variety of problems and the results are compared. Here, the number of nodes in hidden layer has been fixed according to the degree of polynomial in the regression fitting. For this, the input and output data are fitted first with various degree polynomials using regression analysis and the coefficients involved are taken as initial weights to start with the neural training. Fixing of the hidden nodes depends...

View PDFchevron_right
Applications of Artificial Neural Networks to Solve Ordinary Differential Equations
IJRASET Publication

International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2022

Applications of Neural Networks to numerical problems have gained increasing interest. Solving Ordinary Differential Equations can be realized with simple artificial neural network architectures. In this paper, first step is training a Neural Network to satisfy the condition required by a differential equation and then finding a function whose derivative satisfies the Ordinary Differential Equation condition. The method of solving differential equation is implemented in Python Programming using the TensorFlow library.

View PDFchevron_right
Regression-based neural network training for the solution of ordinary differential equations
Snehashish Chakraverty

International Journal of Mathematical Modelling and Numerical Optimisation, 2013

In this paper, we have introduced a method which is based on the use of unsupervised type of regression-based algorithm (RBA) for solving ordinary differential equations (ODEs) with initial or boundary conditions. Approximate solution of differential equation is differentiable and closed analytic. Here we have used error back propagation method for minimising the error function and modification of the parameters without direct use of other optimisation techniques. Initial weights are taken as combination of random as well as by proposed regression-based method. We present the method for solving a variety of problems and the results with arbitrary and regression-based initial weights are compared. Here the number of nodes in hidden layer has been fixed according to the degree of polynomial in the regression. The present model demonstrates to get also the approximate solutions for the differential equation inside and outside of the training domain.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved