Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Conclusion
References
All Topics
Computer Science
Artificial Intelligence

Modular search space for automated design of neural architecture

Profile image of Pavlo RadiukPavlo Radiuk

2020, Proceedings of the O.S. Popov ОNAT

https://doi.org/10.33243/2518-7139-2020-1-1-37-44
visibility

…

description

8 pages

link

1 file

Abstract

The past years of research have shown that automated machine learning and neural architecture search are an inevitable future for image recognition tasks. In addition, a crucial aspect of any automated search is the predefined search space. As many studies have demonstrated, the modularization technique may simplify the underlying search space by fostering successful blocks' reuse. In this regard, the presented research aims to investigate the use of modularization in automated machine learning. In this paper, we propose and examine a modularized space based on the substantial limitation to seeded building blocks for neural architecture search. To make a search space viable, we presented all modules of the space as multisectoral networks. Therefore, each architecture within the search space could be unequivocally described by a vector. In our case, a module was a predetermined number of parameterized layers with information about their relationships. We applied the proposed modular search space to a genetic algorithm and evaluated it on the CIFAR-10 and CIFAR-100 datasets based on modules from the NAS-Bench-201 benchmark. To address the complexity of the search space, we randomly sampled twenty-five modules and included them in the database. Overall, our approach retrieved competitive architectures in averaged 8 GPU hours. The final model achieved the validation accuracy of 89.1% and 73.2% on the CIFAR-10 and CIFAR-100 datasets, respectively. The learning process required slightly fewer GPU hours compared to other approaches, and the resulting network contained fewer parameters to signal lightness of the model. Such an outcome may indicate the considerable potential of sophisticated ranking approaches. The conducted experiments also revealed that a straightforward and transparent search space could address the challenging task of neural architecture search. Further research should be undertaken to explore how the predefined knowledge base of modules could benefit modular search space. Анотація. За минулі роки дослідження підтвердили, що автоматизоване машинне навчання та пошук архітектури нейронної мережі-це неминуче майбутнє для завдань розпізнавання зображень. Крім того, одним із вирішальних аспектів будь-якого автоматизованого пошуку виявився попередньо визначений простір пошуку. Як показали багато обчислювальних досліджень, техніка модуляризації здатна спростити базовий простір пошуку, сприяючи повторному використанню успішних блоків. У зв'язку з цим, ця наукова стаття має на меті дослідити використання модуляризації в автоматизованому машинному навчанні. У цій статті ми пропонуємо та оцінюємо модульований простір, з огляду на істотне обмеження попередньо визначених блоків для пошуку архітектури. Щоб зробити простір пошуку істотним, ми показали всі модулі простору, як багато секторальні мережі. Тому кожну архітектуру в просторі пошуку однозначно описано вектором. У нашому випадку модуль є Radiuk P.M. Modular search space for automated design of neural architecture 37

Figures (4)
As seen from Table 1, our approach’s validation accuracy, which is 89,1%, is below the competition, yet remains efficient considering the search time of almost 8 GPU hours and the restricted number of modules. Also, it is worth noting the potential impact of small search spaces with learned ratings on performance. Using a few ranked modules can result in increased productivity. Table 2 introduces the computational results of CIFAR-100.
As seen from Table 1, our approach’s validation accuracy, which is 89,1%, is below the competition, yet remains efficient considering the search time of almost 8 GPU hours and the restricted number of modules. Also, it is worth noting the potential impact of small search spaces with learned ratings on performance. Using a few ranked modules can result in increased productivity. Table 2 introduces the computational results of CIFAR-100.
Results and discussion. The results of the examination based on the random initialization of 5 modules are presented herein. In addition to examining the proposed modular search space, we compared it to the two most recognized approaches in the search space designing. Table 1 shows the results of CIFAR-10.
Results and discussion. The results of the examination based on the random initialization of 5 modules are presented herein. In addition to examining the proposed modular search space, we compared it to the two most recognized approaches in the search space designing. Table 1 shows the results of CIFAR-10.

Related papers

ОСОБЛИВОСТІ НЕЙРОМЕРЕЖЕВОГО МОДЕЛЮВАННЯ ПРОЦЕСІВ РОЗПІЗНАВАННЯ ЛІНГВІСТИЧНИХ МАРКЕРІВ КАТЕГОРІЙ СМИСЛУ ТА АБСУРДУ В КОНТЕКСТІ РОБОТИ З НЕПРАВДИВИМИ ДАНИМИ
Oleksii Dovhan

Akademìčnì studìï. Serìâ Gumanìtarnì nauki, 2023

The full-scale invasion of Ukraine by the russian federation had a number of geopolitical and domestic consequences. One of the manifestations of such consequences on both of the above levels was the change in the modern information environment under the influence of the actualization of numerous russian narratives. Naturally, the consequence of such false information should be the weakening and vulnerability of Ukrainians to manipulation, disinformation, etc. The latter is represented in the peculiarities of updating the above-mentioned data with false, exaggerated, distorted, etc. information. The above, in turn, determines the relevance and urgency of the problem of their existence in the context of russian disinformation, misinformation propaganda, and fakes (as well as diplomatic fakes), as well as the development of tools for their analysis to prevent their destructive impact. We are talking about the nature of the language poly system and Natural Language Processing (hereinafter -NLP) as tools for processing language data for further use in the process of Machine Learning (hereinafter -ML) and analysis using an artificial neural network. These include vectorization, tokenization, lemmatization, speech interface (Google Assistant and others), automatic translation (Google Translate, Reverso, DeepL, etc.), and other areas. Naturally, the aforementioned study is of particular relevance in the context of the hybrid nature of the Russian-Ukrainian war, in which the media space (online discourse) is home to a whole bunch of fake, distorted, actually false, and other data. That is why in our research we will analyze the main issues related to the problem of methodological features of neural network modeling of the processes of recognizing linguistic markers of the categories of sense and absurdity, paying attention to their nature, mutual influence, and interdependence, priority and socio-cultural significance for Ukrainian society. In addition, we will consider the specifics of the above process in the context of working with false data (disinformation), as well as the impact of the latter on its course.

View PDFchevron_right
Neural Architecture Search: A Visual Analysis
Gabriela Ochoa

Lecture Notes in Computer Science, 2022

Neural architecture search (NAS) refers to the use of search heuristics to optimise the topology of deep neural networks. NAS algorithms have produced topologies that outperform human-designed ones. However, contrasting alternative NAS methods is difficult. To address this, several tabular NAS benchmarks have been proposed that exhaustively evaluate all architectures in a given search space. We conduct a thorough fitness landscape analysis of a popular tabular, cell-based NAS benchmark. Our results indicate that NAS landscapes are multi-modal, but have a relatively low number of local optima, from which it is not hard to escape. We confirm that reducing the noise in estimating performance reduces the number of local optima. We hypothesise that localsearch based NAS methods are likely to be competitive, which we confirm by implementing a landscape-aware iterated local search algorithm that can outperform more elaborate evolutionary and reinforcement learning NAS methods.

View PDFchevron_right
Neural Networks in Semantic Analysis
Alexey N . Averkin

2020

This paper presents research of the possibilities of application deep neural networks in semantic analysis. This paper presents the current situation in this area and the prospects for application an artificial intelligence in semantic analysis and trend and tendencies of this science area. For better understanding future tendencies of researches in semantical area we present detailed review of the studies in semantic analysis with using artificial intelligence, studies about a human brain. Keywords-Semantic Analysis, Deep Neural Networks, Forecasting, Processing of Natural Language. Нейронные сети в семантическом анализе Аверкин А.Н. Ярушев С.А. В статье описываются глубинные архитектуры искусственных нейронных сетей и возможности их применения в семантическом анализе. Рассматривается история разработки глубинных нейронных сетей. Исследуются современные тенденции в задачах семантического анализа, а также представляется кратких обзор исследований в данной области.

View PDFchevron_right
A Surgery of the Neural Architecture Evaluators
Xuefei Ning

2020

Neural architecture search (NAS) has recently received extensive attention due to its effectiveness in automatically designing effective neural architectures. A major challenge in NAS is to conduct a fast and accurate evaluation (i.e., performance estimation) of neural architectures. Commonly used fast architecture evaluators include parameter-sharing ones and predictor-based ones. Despite their high evaluation efficiency, the evaluation correlation (especially of the well-performing architectures) is still questionable. In this paper, we conduct an extensive assessment of both the parameter-sharing and predictor-based evaluators on the NAS-Bench-201 search space, and break up how and why different configurations and strategies influence the fitness of the evaluators. Specifically, we develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages. And based on the findings of our experiments, we give pieces of knowledge...

View PDFchevron_right
Франків Р., Лисенко О. (2011 ) Адаптивність та інформаційність як основоположні категорії сучасного дизайну. Співіснування та взаємодія; Вісник харківської державної академії дизайну та мистецтв – Х.: ХДАДМ, №1. –С.63-68 .pdf
Roman Frankiv

The article considers the problems of establishing a new forms in design associated with information society in aspects of interaction between different morphological systems – 3d adaptive and 2d informational.

View PDFchevron_right
Selection of Neural Architecture and the Environment Complexity
Genci Capi

In this paper we consider how the complexity of evolved neural controllers depends on the environment using foraging behavior of the Cyber Rodent in two different environments. In the first environment, each fruit can be seen from limited directions and different groups of fruits become ripe in different periods. In the second environment, fruits inside a zone are rewarding and those outside are aversive. After evolution, agents with recurrent neural controller outperformed those with feed-forward controllers by effectively using the memory of border passage. Simulation and experimental results with the Cyber Rodent robot confirmed the selection of appropriate complexity of neural controller, both in size and structure, through evolution.

View PDFchevron_right
Modularity in neural computing
Wilson Wen

Proceedings of the IEEE, 1999

This paper considers neural computing models for information processing in terms of collections of subnetwork modules. Two approaches to generating such networks are studied. The first approach includes networks with functionally independent subnetworks, where each subnetwork is designed to have specific functions, communication, and adaptation characteristics. The second approach is based on algorithms that can actually generate network and subnetwork topologies, connections, and weights to satisfy specific constraints. Associated algorithms to attain these goals include evolutionary computation and selforganizing maps. We argue that this modular approach to neural computing is more in line with the neurophysiology of the vertebrate cerebral cortex, particularly with respect to sensation and perception. We also argue that this approach has the potential to aid in solutions to large-scale network computational problems-an identified weakness of simply defined artificial neural networks.

View PDFchevron_right
NETARCH - Exploring architecture search for Neural Networks.pdf
Noémien Kocher

Architecture search for Neural Networks is today one of the major challenges to achieving relevant and always more competitive machine learning models. Choosing the structure of Neural Networks with their multiple layers, learning algorithms, cost functions and learning parameters is more an art that involves some intuition and expertise in the field than a formal process. While some algorithms and heuristics such as grid search or Bayesian approaches can help this process, Neural Networks still lack a comprehensive tool to express such search spaces where the intrinsic structure and not only hyper-parameters of a Neural Network can be expressed, traveled and evaluated. In this work, we look at a new promising framework that should allow one to address this problematic of architecture search. For our experiments, we use it with Convolutional Neural Networks to perform documents categorization. On chapter 1 we first define the context of this project along with its objectives and plan. Chapter 2 talks about DeepArchitect, a new framework made to express a Neural Network architecture search space. In the following chapter 3, we study the state-of-the-art regarding documents categorization with Convolutional Neural Networks; this is the task used in our implementation. Chapter 4 and 5 talk about 2 implementations and their results: one using DeepArchitect and performing a simple architecture search, and the other, implementing a state-of-the-art documents cat- egorization model using a more complex Convolutional Neural Network. The last part, chapter 6, ends this work with a summary, conclusion and perspectives. Keywords. Architecture search, Convolutional Neural Networks, Natural Language Processing

View PDFchevron_right
Systematic Review on Neural Architecture Search
Nathan Eskue

Research Square (Research Square), 2024

Machine Learning (ML) has revolutionized various fields, enabling the development of intelligent systems capable of solving complex problems. However, the process of manually designing and optimizing ML models is often timeconsuming, labor-intensive, and requires specialized expertise. To address these challenges, Automatic Machine Learning (AutoML) has emerged as a promising approach that automates the process of selecting and optimizing ML models. Within the realm of AutoML, Neural Architecture Search (NAS) has emerged as a powerful technique that automates the design of neural network architectures, the core components of ML models. It has recently gained significant attraction due to its capability to discover novel and efficient architectures that surpass human-designed counterparts. This manuscript aims to present a systematic review of the literature on this topic published between 2017 and 2023 to identify, analyze, and classify the different types of algorithms developed for NAS. The methodology follows the guidelines of Systematic Literature Review (SLR) methods. Consequently, this study identified 160 articles that provide a comprehensive overview of the field of NAS, encompassing discussion on current works, their purposes, conclusions, and predictions of the direction of this science branch in its main core pillars: Search Space (SSp), Search Strategy (SSt), and Validation Strategy (VSt). Subsequently, the key milestones and advancements that have shaped the field are highlighted. Moreover, we discuss the challenges and open issues that remain in the field. We envision that NAS will continue to play a pivotal role in the advancement of ML, enabling the development of more intelligent and efficient ML models for a wide range of applications.

View PDFchevron_right
SOFTWARE TOOLS FOR ВІМ ANALYSIS AND NEURAL NETWORKS OF ARTIFICIAL INTELLIGENCE ON ITS BASIS
Oleksii Levchenko
View PDFchevron_right

References (42)

  1. Elsken, Thomas, et al. "Neural architecture search: A survey." Journal of Machine Learning Research, vol. 20, no. 55, 2019, pp. 1-21.
  2. Ren, Pengzhen, et al. A comprehensive survey of neural architecture search: Challenges and solutions. Preprint, submitted 1 June 2020, arXiv.org.
  3. Elsken, Thomas, et al. "Efficient multiobjective neural architecture search via Lamarckian evolution." Proceedings of the 7th International Conference on Learning Representations, ICLR-2019, Preprint, submitted 24 April 2018, arXiv.org.
  4. Wong, Catherine, et al. "Transfer learning with neural AutoML." Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS-2018, edited by Samy Bengio and Hanna M Wallach, Curran Associates Inc., 2018, pp. 8366-75.
  5. Veniat, Tom, and Ludovic Denoyer. "Learning time/memory-efficient deep architectures with budgeted super networks." Proceedings of the 31st 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR-2018, IEEE Inc., 2018, pp. 3492-500, DOI:10.1109/CVPR.2018.00368
  6. Tan, Mingxing, and Quoc Le. "EfficientNet: Rethinking model scaling for convolutional neural networks." Proceedings of the 36th International Conference on Machine Learning, edited by Kamalika Chaudhuri and Ruslan Salakhutdinov, ICML-2019, vol. 97, PMLR.org, 2019, pp. 6105-14.
  7. Dong, Xuanyi, and Yi Yang. "NAS-Bench-201: Extending the scope of reproducible neural architecture search." Proceedings of the 8th International Conference on Learning Representations, ICLR- 2020, Preprint, submitted 15 Jan. 2020, arXiv.org.
  8. Real, Esteban, et al. "Regularized evolution for image classifier architecture search." Proceedings of the 2019 AAAI Conference on Artificial Intelligence, IAAI-2019, vol. 33, no. 1, AAAI Press., 2019, pp. 4780-89, DOI:10.1609/aaai.v33i01.33014780.
  9. Zoph, B., et al. "Learning transferable architectures for scalable image recognition." Proceedings of the 31st 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR-2018, IEEE Inc., 2018, pp. 8697-710, D0I:10.1109/CVPR.2018.00907.
  10. Bello, Irwan, et al. "Neural optimizer search with reinforcement learning." Proceedings of the 34th International Conference on Machine Learning, ICML-2017, edited by Doina Precup and Yee Whye Teh, JMLR.org, 2017, pp. 459-468, D0I:10.5555/3305381.3305429
  11. Pham, Hieu, et al. "Efficient neural architecture search via parameters sharing." Proceedings of the 35th International Conference on Machine Learning, ICML-2018, edited by Jennifer Dy and Andreas Krause, vol. 80, JMLR.org, 2018, pp. 4095-104.
  12. Zhang, Xingcheng, et al. "PolyNet: A pursuit of structural diversity in very deep networks." Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR-2017, vol. 2017-Janua, IEEE Inc., 2017, pp. 3900-08, D0I:10.1109/CVPR.2017.415.
  13. Zhong, Zhao, et al. "Practical block-wise neural network architecture generation." Proceedings of the 31st 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR-2018, IEEE Inc., 2018, pp. 2423-32, D0I:10.1109/CVPR.2018.00257.
  14. Negrinho, Renato, et al. "Towards modular and programmable architecture search." Proceedings of the 33rd Conference on Neural Information Processing Systems, NIPS-2019, edited by H Wallach et al., Curran Associates Inc., 2019, pp. 13715-25.
  15. Wei, Tao, et al. "Modularized morphing of deep convolutional neural networks: A graph approach." IEEE Transactions on Computers, Preprint, submitted 16 April 2020, IEEE Inc., 2020, D0I:10.1109/TC.2020.2988006.
  16. Krizhevsky, Alex. "Learning multiple layers of features from tiny images."Master's Thesis, The University of Toronto, 2009. 60 p. 17. Romanuke, Vadim. "An attempt of finding an appropriate number of convolutional layers in CNNs based on benchmarks of heterogeneous datasets." Electrical, Control and Communication Engineering, vol. 14, no. 1, Sciendo, 2018, pp. 51-57, DOI:10.2478/ecce-2018-0006
  17. Romanuke, Vadim. "Appropriate number of standard 2 * 2 max pooling layers and their allocation in convolutional neural networks for diverse and heterogeneous datasets." Information Technology and Management Science, vol. 20, no. 1, Sciendo, 2017, pp. 12-19, D0I:10.1515/itms-2017-0002
  18. Radiuk, Pavlo. "Application of a genetic algorithm to search for the optimal convolutional neural network architecture." Herald of Khmelnytskyi National University, vol. 281, no. 1, 2020, pp. 7-11, D0I:10.31891/2307-5732-2020-281-1-7-11.
  19. Radiuk, Pavlo. "Impact of training set batch size on the performance of convolutional neural networks for diverse datasets." Information Technology and Management Science, vol. 20, no. 1, Sciendo, 2017, pp. 20-24, D0I:10.1515/itms-2017-0003. Наукові праці ОНАЗ ім. О.С. Попова, 2020, № 1 ЛІТЕРАТУРА:
  20. Elsken T. Neural architecture search: A survey / T. Elsken, J. H. Metzen, F. Hutter // Journal of Machine Learning Research. -2019. -Vol. 20, No. 55. -P. 1-21.
  21. Ren P. A comprehensive survey of neural architecture search: Challenges and solutions / P. Ren, Y. Xiao, X. Chang // Preprint. -2020.
  22. Elsken T. Efficient multiobjective neural architecture search via Lamarckian evolution / T. Elsken, J. H. Metzen, F. Hutter // Proceedings of the 7th International Conference on Learning Representations, ICLR-2019, New Orleans, LA, USA, 6-9 May 2019. -P. 1-23.
  23. Wong C. Transfer learning with neural AutoML / C. Wong, N. Houlsby, Y. Lu // Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS-2018, Montréal, QC, Canada, 2-8 Dec. 2018, Curran Associates Inc. -P. 8366-8375.
  24. Veniat T. Learning time/memory-efficient deep architectures with budgeted super networks / T. Veniat, L. Denoyer // Proceedings of the 31st 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR-2018, Salt Lake City, UT, USA, 18-23 June 2018, IEEE Inc. -P. 3492-3500. D0I:10.1109/CVPR.2018.00368
  25. Tan M. EfficientNet: Rethinking model scaling for convolutional neural networks / M. Tan, Q. Le // Proceedings of the 36th International Conference on Machine Learning, Long Beach, California, USA, PMLR.org. -P. 6105-6114.
  26. Dong X. NAS-Bench-201: Extending the scope of reproducible neural architecture search / X. Dong, Y. Yang // Proceedings of the 8th International Conference on Learning Representations, ICLR- 2020, Addis Ababa, Ethiopia, 26 April -1 May 2020. -P. 1 -16.
  27. Real E. Regularized evolution for image classifier architecture search / E. Real, A. Aggarwal, Y. Huang // Proceedings of the AAAI Conference on Artificial Intelligence, IAAI-2019, Honolulu, HI, USA, 27
  28. January -1 February 2019, AAAI Press. -P. 4780-4789. D0I:10.1609/aaai.v33i01.33014780.
  29. Zoph B. Learning transferable architectures for scalable image recognition / B. Zoph, V. Vasudevan, J. Shlens // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR-2018, Salt Lake City, UT, USA, 18-23 June 2018. -P. 8697-8710. D0I:10.1109/CVPR.2018.00907. 10. Bello I. Neural optimizer search with reinforcement learning / I. Bello, B. Zoph, V. Vasudevan // Proceedings of the 34th International Conference on Machine Learning, ICML-2017, Sydney, Australia, 7-11 Aug. 2017, JMLR.org. -P. 459-468. D0I:10.5555/3305381.3305429
  30. Pham H. Efficient neural architecture search via parameters sharing / H. Pham, M. Guan, B. Zoph // Proceedings of the 35th International Conference on Machine Learning, ICML-2018, Stockholmsmassan, Stockholm Sweden, 11-15 July 2018, JMLR.org. -P. 4095-4104.
  31. Zhang X. PolyNet: A pursuit of structural diversity in very deep networks / X. Zhang, Z. Li, C. Loy // Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR-2017, Honolulu, HI, USA, 21-26 July 2017, IEEE Inc. -P. 3900-3908. D0I:10.1109/CVPR.2017.415. 13. Zhong Z. Practical block-wise neural network architecture generation / Z. Zhong, J. Yan, W. Wu //
  32. IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE Inc. -P. 2423-2432. D0I:10.1109/CVPR.2018.00257.
  33. Negrinho R. Towards modular and programmable architecture search / R. Negrinho, M. Gormley, G. J. Gordon // Proceedings of the 33rd Conference on Neural Information Processing Systems, NIPS-2019, Vancouver, BC, Canada, 8-14 Dec. 2019, Curran Associates Inc. -P. 13715-13725.
  34. Wei T. Modularized morphing of deep convolutional neural networks: A graph approach / T. Wei, C. Wang, C. W. Chen // IEEE Transactions on Computers, Preprint. -2020. D0I:10.1109/TC.2020.2988006.
  35. Krizhevsky, A. Learning multiple layers of features from tiny images: master's thesis / A. Krizhevsky // The University of Toronto. -60 p. 17. Romanuke V. V. An attempt of finding an appropriate number of convolutional layers in CNNs based on benchmarks of heterogeneous datasets / V. V. Romanuke // Electrical, Control, and Communication Engineering. -2018. -Vol. 14, No. 1. -P. 51-57. D0I:10.2478/ecce-2018-0006.
  36. Radiuk P.M. Modular search space for automated design of neural architecture
  37. Romanuke V. V. Appropriate number of standard 2 * 2 max pooling layers and their allocation in convolutional neural networks for diverse and heterogeneous datasets / V. V. Romanuke // Information Technology and Management Science. -2017. -Vol. 20, No. 1. -P. 12-19. DOI:10.1515/itms-2017-0002
  38. Radiuk P. M. Application of a genetic algorithm to search for the optimal convolutional neural network architecture / P. M. Radiuk // Herald of Khmelnytskyi National University. -2020. -Vol. 281, No. 1.
  39. -P. 7-11. DOI:10.31891/2307-5732-2020-281-1-7-11.
  40. Radiuk P. M. Impact of training set batch size on the performance of convolutional neural networks for diverse datasets / P. M. Radiuk // Information Technology and Management Science. -2017. - Vol. 20, No.
  41. -P. 20-24. D0I:10.1515/itms-2017-0003. DOI 10.33243/2518-7139-2020-1-1-37-44 Наукові праці ОНАЗ ім. О.С. Попова, 2020, № 1
  42. 44 Radiuk P.M. Modular search space for automated design of neural architecture

Related papers

Development of a method for structural optimization of a neural network based on the criterion of resource utilization efficiency
Oleksii Mykhailenko

Eastern-European Journal of Enterprise Technologies, 2019

Для вирішення задач апроксимації широко використовуються математичні моделі у вигляді штучних нейронних мереж (ШНМ). Використання цієї технології передбачає двох етапний підхід. На першому етапі визначається структура моделі ШНМ, а на другому етапі здійснюється навчання для отримання максимального наближення до еталонної моделі. Максимальне значення наближення до еталону визначається складністю архітектури ШНМ. Тобто, підвищення складності моделі ШНМ дозволяє підвищувати точність апроксимації, а, відповідно, і результату навчання. При цьому визначення структури моделі ШНМ, що здійснює апроксимацію із заданою точністю, визначається як процес оптимізації. Однак підвищення складності ШНМ призводить не тільки до підвищення точності, а і до підвищення часу обчислювального процесу. Таким чином, показник «задана точність» не може використовуватися в задачах визначення оптимальної архітектури нейронної мережі. Це пов'язано з тим, що результат вибору структури моделі і процесу її навчання, котрий базується на забезпеченні необхідної точ ності апроксимації, може зайняти неприйнятний для користувача часовий проміжок. Для вирішення завдання структурної ідентифікації нейронної мережі використовується підхід, у рамках якого здійснюється визначення конфігурації моделі за критерієм ефективності. У про цесі реалізації розробленого методу узгоджується часовий чинник вирішення завдання і точністю апроксимації. Запропонований підхід дозволяє обґрунтувати принцип вибору структури і параметрів нейронної мережі, спираючись на максимальне значення показника ефективності використання ресурсів Ключові слова: штучна нейронна мережа, оптимізація структури, апроксимація функцій, критерій ефективності UDC 007.5

View PDFchevron_right
Розробка методики навчання штучних нейронних мереж для інтелектуальних систем підтримки прийняття рішень
Andrii Shyshatskyi

2020

The method of training artificial neural networks for intelligent decision support systems is developed. A distinctive feature of the proposed method is that it provides training not only of the synaptic weights of the artificial neural network, but also the type and parameters of the membership function. If it is impossible to provide the specified quality of functioning of artificial neural networks due to the learning of the parameters of the artificial neural network, the architecture of artificial neural networks is trained. The choice of architecture, type and parameters of the membership function is based on the computing resources of the tool and taking into account the type and amount of information supplied to the input of the artificial neural network. Due to the use of the proposed methodology, there is no accumulation of errors of training artificial neural networks as a result of processing information that is fed to the input of artificial neural networks. Also, a dis...

View PDFchevron_right
Розробка алгоритму навчання штучних нейронних мереж для інтелектуальних систем підтримки прийняття рішень
Andrii Shyshatskyi

2020

The algorithm to train artificial neural networks for intelligent decision support systems has been constructed. A distinctive feature of the proposed algorithm is that it conducts training not only for synaptic weights of an artificial neural network, but also for the type and parameters of membership function. In case of inability to ensure the assigned quality of functioning of artificial neural networks due to training of parameters of artificial neural network, the architecture of artificial neural networks is trained. The choice of the architecture, type and parameters of membership function occurs taking into consideration the computation resources of the facility and taking into consideration the type and the amount of information entering the input of an artificial neural network. In addition, when using the proposed algorithm, there is no accumulation of an error of artificial neural networks training as a result of processing the information entering the input of artifici...

View PDFchevron_right
Application of a Genetic Algorithm to Search for the Optimal Convolutional Neural Network Architecture with Weight Distribution
Pavlo Radiuk

Herald of Khmelnytskyi national university, 2020

In the past decade, a new way in neural networks research called Network architectures search has demonstrated noticeable results in the design of architectures for image segmentation and classification. Despite the considerable success of the architecture search in image segmentation and classification, it is still an unresolved and urgent problem. Moreover, the neural architecture search is also a highly computationally expensive task. This work proposes a new approach based on a genetic algorithm to search for the optimal convolutional neural network architecture. We integrated a genetic algorithm with standard stochastic gradient descent that implements weight distribution across all architecture solutions. This approach utilises a genetic algorithm to design a sub-graph of a convolution cell, which maximises the accuracy on the validation set. We show the performance of our approach on the CIFAR-10 and CIFAR-100 datasets with a final accuracy of 93.21% and 78.89%, respectively. The main scientific contribution of our work is the combination of genetic algorithm with weight distribution in the architecture search tasks that achieve similar to state-of-the-art results on a single GPU. Keywords: convolutional neural networks, genetic algorithms, weight distribution, ablation study. П.М. РАДЮК Хмельницький національний університет ЗАСТОСУВАННЯ ГЕНЕТИЧНОГО АЛГОРИТМУ ДЛЯ ПОШУКУ ОПТИМАЛЬНОЇ АРХІТЕКТУРИ ЗГОРТКОВОЇ НЕЙРОННОЇ МЕРЕЖІ З РОЗПОДІЛЕННЯМ ВАГ За останнє десятиліття новий спосіб дослідження нейронних мереж під назвою «Пошук мережевих архітектур» продемонстрував позитивні результати в розробці архітектур для сегментації та класифікації зображень. Незважаючи на значний успіх пошуку архітектур в задачах сегментації та класифікації зображень, він все ще є невирішеною і актуальною проблемою. Більше того, пошук архітектур нейронних мереж є також дуже витратим з точки зору обчислювальних ресурсів. У цій роботі пропонується новий підхід на основі генетичного алгоритму для пошуку оптимальної архітектури згорткової нейронної мережі. Ми інтегрували генетичний алгоритм зі стандартним стохастичним градієнтом, що реалізує розподіл ваг у всіх архітектурних рішеннях. Цей підхід використовує генетичний алгоритм для проектування частини графу в якості згорткового шару, що забезпечує максимальну точність на валідаційному наборі даних. У цій роботі ми демонструємо ефективність нашого підходу на наборах даних CIFAR-10 та CIFAR-100 з кінцевою точністю 93,21 % та 78,89 % відповідно. Основним науковим внеском нашої роботи є поєднання генетичного алгоритму з розподілом ваг в задачах пошуку архітектури, що досягає точності класифікацїі зображення з використанням одного графічного процесора близької до найсучасніших результатів. Ключові слова: згорткові нейронні мережі, генетичні алгоритми, розподілення ваг, абляція дослідження. Introduction In recent decades, artificial neural networks have produced outstanding results in computer vision with a wide variety of applied tasks such as object detection, image segmentation and classification and others. The design of neural network architectures for a specific task or dataset usually requires specific approaches and a large number of computational resources [1, 2]. Recently, a new way in neural networks research called Network Architectures Search (NAS) has demonstrated noticeable results in the design of architectures for image segmentation and classification. NAS approaches use a recurrent neural network (RNN) controller to generate a candidate network architecture, called child model, which is then trained to converge. After the training, researches measure the performance of the trained network architecture on the desired task or dataset. RNN controller receives the performance measurement as a signal to explore for a better architecture. After that, this process repeats over many computationally expensive iterations. Analysis of recent research Despite the considerable success of NAS in classification tasks, it is still highly computationally expensive. Recently, many studies have used the idea of parameter sharing [3] across all child models to reduce the need for training each child model from scratch, thereby eliminating most computational costs. Weight distribution, the analogue of parameter sharing for convolutional cells, has shown prominent result utilising reinforcement-based and gradient-based methods. The first method is an effective search for neural architecture using parameter sharing (ENAS) [4] based on reinforcement training with the RNN controller to create the candidate architecture. The second method is called a differentiable architecture search with various modifications (DARTS) [5], where each compound has a gradient-updating probability function. However, none of the publications has yet combined GAs with parameter sharing or its analogies to NAS. Genetic algorithm (GA) is a search method based on natural selection and genetics [6]. GAs consist of four fundamental concepts: selection, cross-over, mutation, and replacement. Population, another essential component of GA, is utilised to generate new candidate solutions. A new generation is created in each iteration, using a three-step process of selection, cross-over and mutation. The next generation is then inserted into the population through the replacement phase. The algorithm starts with a random set that is evaluated at the beginning of training.

View PDFchevron_right
On the Interrelation between Natural and Artificial Intelligence (О соотношении искусственного и естественного интеллекта)
Сергей Катречко

Abstract. My paper is devoted to the problem of conceptual relation between the human mind (intellect) and artificial intelligence. It begins with the thesis that advances in computer sciences allow us to tell about building Artificial Intelligence (in a ‘weak sense’, according to Searle) now which modeling only separate intellectual functions (activity) of the human. However, such AI not neither full-fledged (universal) system, nor the exact model of natural intelligence (human mind). In this regard in article three fundamental differences of human mind from the existing systems of AI are discussed: 1) the mathematical human thinking (the mathematics is understood, according to Kant, as “cognition that from the construction of concepts” [Critique of Pure Reason B741]); 2) the problem of mental images (the presence of mental images in the human mind and its absence (non-obligation) in AI–systems); 3) the consciousness phenomenon: the consciousness as cartesian ‘cogito’ (or Kant’s ‘transcendental apperception’) is inextricably connected with the natural way of thinking, but is an optional component for AI–systems Статья посвящена проблеме концептуального соотношения человеческого ума (интеллекта) и искусственного интеллекта. Первоначальным выступает тезис о том, что достижения в области computer sciences позволяют говорить о построении в настоящее время Искусственного Интеллекта (в ‘слабом смысле’, по Сёрлю), который моделирует отдельные интеллектуальные функции (деятельности) человека. Однако такой ИИ не является ни полноценной (универсальной) системой, ни точной копией естественного интеллекта (человеческого ума). В этой связи в статье обсуждаются три принципиальных отличия человеческого ума от существующих систем ИИ: 1) математическое мышление человека (математика понимается, по Канту, как «познание посредством конструирования [конструкции из] понятий» [Критика чистого разума, В741]); 2) проблема ментальных образов (наличие ментальных образов в человеческом уме и отсутствие (необязательность) таковых в системах ИИ); 3) феномен сознания (декартовское cogito, кантовская трансцендентальная апперцепция), без которого нельзя помыслить человеческий ум, но которое отсутствует (или является необязательным компонентном) для существующих систем ИИ.

View PDFchevron_right

Related topics

  • Neural Network Architecture
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved