Academia.eduAcademia.edu

Figure 11 – uploaded by Xiangxiang Wang

See full PDFdownloadDownload figure
FIGURE 10. Resnet50: Comparison of training and validation accuracy of our proposed RMAF to two baseline (ReLU and Tanh) activation functions on CIFAR100. (a) Shows training and validation accuracy of ReLU achieving 75.7% and 61.2% respectively, (b) Training and Validation of Tanh had 64.1% and 54.2% respectively. We show that our proposed function (c) achieves higher performance training and validation accuracy (i.e. 79.8% and 66.3%) compared to ReLU (a) and Tanh (b) on CIFAR100 dataset.

Figure 10 Resnet50: Comparison of training and validation accuracy of our proposed RMAF to two baseline (ReLU and Tanh) activation functions on CIFAR100. (a) Shows training and validation accuracy of ReLU achieving 75.7% and 61.2% respectively, (b) Training and Validation of Tanh had 64.1% and 54.2% respectively. We show that our proposed function (c) achieves higher performance training and validation accuracy (i.e. 79.8% and 66.3%) compared to ReLU (a) and Tanh (b) on CIFAR100 dataset.

Related Figures (17)

FIGURE 1. The proposed RMAF function and its derivative. This work has noticed that, the RMAF at x > 0 has a similar property to “Swish” which was proposed recently by Google Brain [19]. Fig. 2 shows plot of RMAF func- tion and derivative with different initialized p. Moreover, we visualized the comparison of RMAF with ReLU and Swish in Fig. 3, which their derivative plots are shown. Swish shows much superiority over ReLU on several deep models in image classification and machine translation problems [19]. Mean- while, the derivative of Swish is composed with high portion of nonsparse property and hence, increase the computational complexity of the given network. However, RMAF can retain its hard zero property which on the other side ReLU which often deactivated most of the neurons during both forward and backward propagation.
FIGURE 1. The proposed RMAF function and its derivative. This work has noticed that, the RMAF at x > 0 has a similar property to “Swish” which was proposed recently by Google Brain [19]. Fig. 2 shows plot of RMAF func- tion and derivative with different initialized p. Moreover, we visualized the comparison of RMAF with ReLU and Swish in Fig. 3, which their derivative plots are shown. Swish shows much superiority over ReLU on several deep models in image classification and machine translation problems [19]. Mean- while, the derivative of Swish is composed with high portion of nonsparse property and hence, increase the computational complexity of the given network. However, RMAF can retain its hard zero property which on the other side ReLU which often deactivated most of the neurons during both forward and backward propagation.
FIGURE 2. RMAF function and its derivative with different p. In (a) the plot represents the RMAF with the parameter p = 1. (b) shows a RMAF function with different selected p. (c) represent the derivative of RMAF, while (d) shows the derivative of RMAF with different p. In this work, all curves were set with the same alpha (i.e. « = 1). Best view in color. Substitute (12) (13) and (14) into equation (15) such that ve can obtain
FIGURE 2. RMAF function and its derivative with different p. In (a) the plot represents the RMAF with the parameter p = 1. (b) shows a RMAF function with different selected p. (c) represent the derivative of RMAF, while (d) shows the derivative of RMAF with different p. In this work, all curves were set with the same alpha (i.e. « = 1). Best view in color. Substitute (12) (13) and (14) into equation (15) such that ve can obtain
FIGURE 3. Visualization of RMAF, its derivative compared with ReLU and Swish activation functions. (a) RMAF vs ReLU, (b) The derivative of RMAF and ReLU. (c) RMAF vs Swish and (d) The derivative of RMAF and Swish. The derivative of RMAF is denoted as RMAF’. This is same with the ReLU and Swish as shown at the upper left of each of the figures. Best view in color.
FIGURE 3. Visualization of RMAF, its derivative compared with ReLU and Swish activation functions. (a) RMAF vs ReLU, (b) The derivative of RMAF and ReLU. (c) RMAF vs Swish and (d) The derivative of RMAF and Swish. The derivative of RMAF is denoted as RMAF’. This is same with the ReLU and Swish as shown at the upper left of each of the figures. Best view in color.
FIGURE 4. Sample data of MNIST and CIFAR.(a) Shows a sample data of MNIST, (b) shows sample data of CIFAR-10 and CIFAR-100. We note that, CIFAR-10 and CIFAR-100 are composed of different total class samples. As the names depict, the total classes of CIFAR-10 is 10 while CIFAR-100 has 100 classes. TABLE 1. Database information of Wisconsin breast cancer, MNIST, Iris Car evaluation, CIFAR-10 and CIFAR-100 showing training, testing, total set and the number of classes for each datasets.
FIGURE 4. Sample data of MNIST and CIFAR.(a) Shows a sample data of MNIST, (b) shows sample data of CIFAR-10 and CIFAR-100. We note that, CIFAR-10 and CIFAR-100 are composed of different total class samples. As the names depict, the total classes of CIFAR-10 is 10 while CIFAR-100 has 100 classes. TABLE 1. Database information of Wisconsin breast cancer, MNIST, Iris Car evaluation, CIFAR-10 and CIFAR-100 showing training, testing, total set and the number of classes for each datasets.
FIGURE 5. A structure of our multi-layer perceptron (MLP). network is a deep network due to the number of hidden layers implemented, which composed of more than two hidden layers. Fig. 5 shows our simple design structure of MLP.
FIGURE 5. A structure of our multi-layer perceptron (MLP). network is a deep network due to the number of hidden layers implemented, which composed of more than two hidden layers. Fig. 5 shows our simple design structure of MLP.
FIGURE 8. SqueezeNet model validation accuracy on the (a) CIFAR-10, (b) CIFAR-100, (c) MNIST and (d) ImageNet dataset trained with 200 epochs.
FIGURE 8. SqueezeNet model validation accuracy on the (a) CIFAR-10, (b) CIFAR-100, (c) MNIST and (d) ImageNet dataset trained with 200 epochs.
FIGURE 6. Pre-activated residual module: RMAF and Weight represent activation and convolutional layers. The X; and X;,, are input and output volumes, respectively. Here residual module adds the input volume X, with output volume of the second convoluional layer F(x)+x to obtain the final output volume X;,_;.
FIGURE 6. Pre-activated residual module: RMAF and Weight represent activation and convolutional layers. The X; and X;,, are input and output volumes, respectively. Here residual module adds the input volume X, with output volume of the second convoluional layer F(x)+x to obtain the final output volume X;,_;.
TABLE 2. Classification performance of MLP over Breast cancer, Cars Evaluation, Iris and MNIST datasets for different activations.
TABLE 2. Classification performance of MLP over Breast cancer, Cars Evaluation, Iris and MNIST datasets for different activations.
TABLE 3. Classification performance of ResNet50, Alexnet, Squeezenet and Densenet121 on CIFAR-10, CIFAR-100, MNIST and ImageNet dataset for different activation functions.
TABLE 3. Classification performance of ResNet50, Alexnet, Squeezenet and Densenet121 on CIFAR-10, CIFAR-100, MNIST and ImageNet dataset for different activation functions.
FIGURE 9. Resnet50: Comparison of training and validation accuracy of our proposed RMAF to two baseline (ReLU and Tanh) activation functions on CIFAR10. (a) Training and Validation accuracy of ReLU achieved 95.9% and 89.5% respectively, (b) Training and Validation of Tanh had 92.2% and 86.6% respectively. We show that our proposed function (c) achieves higher performance training and validation accuracy (i.e. 98.7% and 90.3%) compared to ReLU (a) and Tanh (5) on CIFAR10 dataset. TABLE 4. Statistical features metrics on RMAF validation accuracy.
FIGURE 9. Resnet50: Comparison of training and validation accuracy of our proposed RMAF to two baseline (ReLU and Tanh) activation functions on CIFAR10. (a) Training and Validation accuracy of ReLU achieved 95.9% and 89.5% respectively, (b) Training and Validation of Tanh had 92.2% and 86.6% respectively. We show that our proposed function (c) achieves higher performance training and validation accuracy (i.e. 98.7% and 90.3%) compared to ReLU (a) and Tanh (5) on CIFAR10 dataset. TABLE 4. Statistical features metrics on RMAF validation accuracy.
TABLE 5. Statistic features.
TABLE 5. Statistic features.
FIGURE 12. Squeeznet: Comparison of training and validation accuracy of our proposed RMAF with two baseline (ReLU and Tanh) activation functions or CIFAR100 based on Squeezenet. (a) Shows training and validation accuracy of ReLU achieving 84.85% and 82.36% respectively, (b) Training and Validation of Tanh had 80.80% and 81.58% respectively. We show that our proposed function (c) achieves higher performance training and validation accuracy (i.e. 90.50% and 86.45%) compared to (a) and (b) on CIFAR100 dataset. Table 3 presents other variants of ReLU performance.
FIGURE 12. Squeeznet: Comparison of training and validation accuracy of our proposed RMAF with two baseline (ReLU and Tanh) activation functions or CIFAR100 based on Squeezenet. (a) Shows training and validation accuracy of ReLU achieving 84.85% and 82.36% respectively, (b) Training and Validation of Tanh had 80.80% and 81.58% respectively. We show that our proposed function (c) achieves higher performance training and validation accuracy (i.e. 90.50% and 86.45%) compared to (a) and (b) on CIFAR100 dataset. Table 3 presents other variants of ReLU performance.
KWABDENA ADU received the b.s. degree in information technology from the University of Education Winneba—Kumasi, in 2015, and the M.S. degree in software engineering from the University of Electronic Science and Technology of China, in 2018. He is currently pursuing the Ph.D. degree with the School of Information and Software Engineering, University of Electronic Science and Technology of China. His research interests include medical imaging, image process- 1g, computer vision, data science, and machine learning.
KWABDENA ADU received the b.s. degree in information technology from the University of Education Winneba—Kumasi, in 2015, and the M.S. degree in software engineering from the University of Electronic Science and Technology of China, in 2018. He is currently pursuing the Ph.D. degree with the School of Information and Software Engineering, University of Electronic Science and Technology of China. His research interests include medical imaging, image process- 1g, computer vision, data science, and machine learning.
Related topics:
Computer ScienceArtificial IntelligenceSigmoid Function
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved