Taxonomy of neural transfer functions

International Journal of Data Science, 2021

Neural networks have been very important models across computer vision, natural language processing, speech and image recognition, aircraft safety and many more. It uses a variety of architectures that centres on the Multi-Layer Perceptron (MLP) which is the most commonly used type of Artificial Neural Network. MLP has been found to be good in terms of model precision in the usage of Homogenous Transfer/activation Functions (HTFs), especially with large data set. Based on the preliminary investigations of ranking of transfer functions by error variance (Udomboso, 2014), three HTFs are considered to perform better than other HTFs in prediction. These HTFs are the Hyperbolic Tangent Transfer functions (TANH), Hyperbolic Tangent Sigmoid Transfer function (TANSIG) and the Symmetric Saturating Linear Transfer Function (SSLTF). In this work, the performance of two Heterogeneous Transfer Functions (HETFs), which came as a result of the convolution of the three best HTFs, were compared with the performance of the three above listed HTFs. The hidden neurons used are 2, 5 and 10, while the sample sizes include 50, 100, 200, 500 and 1000. The data were divided into training sets of 90, 80 and 70 respectively. The results showed that the HETFs performed better in terms of the forecast using Mean Square Error (MSE), Mean Absolute Error (MAE) and Test Error as the forecast prediction criteria.

The Journal of Supercomputing, 2025

Choosing the optimal activation functions (AFs) for the training of deep neural networks (DNNs) has always posed a significant challenge due to its substantial impact on the network performance and training speed. Despite the impressive performance demonstrated by various nonlinear activation functions, such as rectified linear units, hyperbolic tangent, Sigmoid, Swish, Mish, Smish, and Logish, only a restricted subset of these functions is commonly embraced in most applications, primarily because of inherent inconsistencies or other limitations. This research introduces a novel framework for generating a large family of nonlinear activation functions (LFNAFs) using propositions. These propositions define sufficient conditions for creating continuous and differentiable activation functions with desirable properties, including nonlinearity, zero-centeredness, resilience against vanishing gradients, prevention of gradient saturation, non-monotonicity, smoothness and infinite range. In addition, many well-known activation functions are particular cases of this family, such as ReLU, Logish, Swish and so on. These findings indicate that the activation functions we developed are better suited for simple and intricate deep learning models. The results show that the proposed activation functions achieve significantly better accuracy compared to other well-known activation functions across various datasets, including MNIST, CIFAR-10, and CIFAR-100. Many AFs can be generated, but a few instances of them have been discussed in this paper. For instance, more simulations have been done on H 2 and H 2 +Sigmoid (AFs). Performance of H 2 is obviously better than other AFs, but H 2 +Sigmoid is better and in some cases is equal than other AFs. Notably, while the proposed activation functions achieved 100% training accuracy on the MNIST dataset, it is important to acknowledge that achieving such high accuracy may vary based on the complexity of the dataset and specific conditions of the model applied.

Proceedings of the XII CAEPIA

We present an analysis of the computational capabilities of feed-forward neural networks focusing on the role of the output function. The space of configurations that implement a given target function is analyzed for small size networks when different output functions are considered. The generalization complexity and other relevant properties for some complex and useful linearly separable functions are also analyzed. The results indicate that efficient output functions are those with a similar Hamming weight as the target output that at the same time have a high complexity.

IEEE Transactions on Neural Networks, 1994

Journal of Emerging Technologies and Innovative Research, 2019

From the very start activation functions in the neural network has been a key research area. More and more people are trying to come up with the state of the art activation functions for various purposes. These functions play an important role in deciding how good the model performs. This paper provides an insight into the effects of activation functions on four common datasets. It also shows how the accuracy parameter varies with change the value of the activation function while keeping all other parameters the same. It also lists the best value for activation functions in general for any dataset

Neural Networks, 2021

In the literature, there is a strong interest to identify and define activation functions which can improve neural network performance. In recent years there is a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred as trainable, learnable or adaptable activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss on main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to add neuron layers which use fixed activation functions (nontrainable activation functions) and some simple local rule constrains the corresponding weight layers.

Expert Systems, 1999

This paper examines the capacity of feedforward neural networks (NNs) to approximate certain functional forms. Its purpose is to show that the theoretical property of 'universal approximation', which provides the basic rationale behind the NN approach, should not be interpreted too literally. The most important issue considered involves the number of hidden layers in the network. We show that for a number of interesting functional forms better generalization is possible with more than one hidden layer, despite theoretical results to the contrary. Our experiments constitute a useful set of counter-examples.

Journal of Economic Dynamics & Control, 2003

A limited number of activation functions have been utilized in practice for feedforward artiÿcial neural networks. A class of sigmoidal functions is deÿned and another function, which is the envelope of the derivatives of the members of the deÿned class, is shown to be sigmoidal. The functions deÿned are shown to satisfy the requirements of the universal approximation theorem(s).

Journal of Physics: Conference Series, 2019

The development of Artificial Neural Networks (ANNs) has achieved a lot of fruitful results so far, and we know that activation function is one of the principal factors which will affect the performance of the networks. In this work, the role of many different types of activation functions, as well as their respective advantages and disadvantages and applicable fields are discussed, so people can choose the appropriate activation functions to get the superior performance of ANNs.