Neural Network Architecture

For the past years, many researchers and engineers have been developing and optimising deep neural networks (DNN). The process of neural architecture design and tuning its hyperparameters remains monotonous, timeconsuming, and do not always ensure optimal results. In his regard, the automatic design of machine learning (AutoML) has been widely utilised, and neural architecture search (NAS) has been actively developing in recent years. Despite meaningful advances in the field of NAS, a unified, systematic approach to explore and compare search methods has not been established yet. In this paper, we aim to close this knowledge gap by summarising search decisions and strategies and propose a schematic framework. It applies quantitative and qualitative metrics for prototyping, comparing, and benchmarking the NAS methods. Moreover, our framework enables categorising critical areas to search for better neural architectures.

A problem of improving the performance of convolutional neural networks is considered. A parameter of the training set is investigated. The parameter is the batch size. The goal is to find an impact of training set batch size on the performance. To get consistent results, diverse datasets are used. They areMNIST and CIFAR-10. Simplicity of the MNIST dataset stands against complexity of the CIFAR-10 dataset, although the simpler dataset has 10 classes as well as the more complicated one. To achieve acceptable testing results, various convolutional neural network architectures are selected for the MNIST and CIFAR-10 datasets, with two and five convolutional layers, respectively. The assumption about the dependence of the recognition accuracy on the batch size value is confirmed: the larger the batch size value, the higher the recognition accuracy. Another assumption about the impact of the type of the batch size value on the CNN performance is not confirmed.

The past years of research have shown that automated machine learning and neural architecture search are an inevitable future for image recognition tasks. In addition, a crucial aspect of any automated search is the predefined search space. As many studies have demonstrated, the modularization technique may simplify the underlying search space by fostering successful blocks' reuse. In this regard, the presented research aims to investigate the use of modularization in automated machine learning. In this paper, we propose and examine a modularized space based on the substantial limitation to seeded building blocks for neural architecture search. To make a search space viable, we presented all modules of the space as multisectoral networks. Therefore, each architecture within the search space could be unequivocally described by a vector. In our case, a module was a predetermined number of parameterized layers with information about their relationships. We applied the proposed modular search space to a genetic algorithm and evaluated it on the CIFAR-10 and CIFAR-100 datasets based on modules from the NAS-Bench-201 benchmark. To address the complexity of the search space, we randomly sampled twenty-five modules and included them in the database. Overall, our approach retrieved competitive architectures in averaged 8 GPU hours. The final model achieved the validation accuracy of 89.1% and 73.2% on the CIFAR-10 and CIFAR-100 datasets, respectively. The learning process required slightly fewer GPU hours compared to other approaches, and the resulting network contained fewer parameters to signal lightness of the model. Such an outcome may indicate the considerable potential of sophisticated ranking approaches. The conducted experiments also revealed that a straightforward and transparent search space could address the challenging task of neural architecture search. Further research should be undertaken to explore how the predefined knowledge base of modules could benefit modular search space. Анотація. За минулі роки дослідження підтвердили, що автоматизоване машинне навчання та пошук архітектури нейронної мережі-це неминуче майбутнє для завдань розпізнавання зображень. Крім того, одним із вирішальних аспектів будь-якого автоматизованого пошуку виявився попередньо визначений простір пошуку. Як показали багато обчислювальних досліджень, техніка модуляризації здатна спростити базовий простір пошуку, сприяючи повторному використанню успішних блоків. У зв'язку з цим, ця наукова стаття має на меті дослідити використання модуляризації в автоматизованому машинному навчанні. У цій статті ми пропонуємо та оцінюємо модульований простір, з огляду на істотне обмеження попередньо визначених блоків для пошуку архітектури. Щоб зробити простір пошуку істотним, ми показали всі модулі простору, як багато секторальні мережі. Тому кожну архітектуру в просторі пошуку однозначно описано вектором. У нашому випадку модуль є Radiuk P.M. Modular search space for automated design of neural architecture 37

for generic signal processing applications. In the proposed paper analog components like Gilbert Cell Multiplier (GCM), Neuron activation Function (NAF) are used to implement artificial NNA. The analog components used are comprises of multipliers and adders' along with the tan-sigmoid function circuit using MOS transistor in subthreshold region. This neural architecture is trained using Back propagation (BP) algorithm in analog domain with new techniques of weight storage. Layout design and verification of the proposed design is carried out using Tanner EDA 14.1 tool and synopsys Tspice. The technology used in designing the layouts is MOSIS/HP 0.5u SCN3M, Tight Metal.

Convolutional neural networks (CNNs) have taken the spotlight in a variety of machine learning applications. To reach the desired performance, CNNs have become increasingly deeper and larger which goes along with a tremendous amount of power and storage requirements. Such increases in computational demands and memory make deep learning prohibitive for resource-constrained hardware platforms such as mobile devices. In order to address this problem, we provide a general framework which renders efficient CNN representations that solve given classification tasks to specified quality levels. More precisely, the framework yields sparse and ternary neural networks, i.e. networks with many parameters set to zero and the non-zero parameters quantized from 32 bit to 2 bit. Ternary networks are not only efficient in terms of storage but also in terms of computational complexity. By explicitly boosting sparsity we reach further efficiency gains. The proposed framework follows a two-step paradigm. First, a baseline model is extended by compound model scaling until a specified target accuracy is reached. Secondly, our Entropy-Constrained Trained Ternarization (EC2T) algorithm is applied which simultaneously quantizes and sparsifies the scaled model. Here, a λ-operator balances the entropy constraint and thus the compression gain of the resulting network. We validated the effectiveness of EC2T in a variety of experiments. This includes CIFAR-10, CIFAR-100 and ImageNet classification tasks and the compression of renowned architectures (i.e., ResNets and EfficientNet) as well as our own compound-scaled models. We show the advantages of EC2T compared to the standard in ternary quantization, Trained Ternary Quantization (TTQ), and set new benchmarks in this research area. For instance, EC2T compresses ResNet-20 by more than 24x and reduces the number of arithmetical operations by more than 12x while causing a minimal accuracy degradation of 0.9%.

In the past decade, a new way in neural networks research called Network architectures search has demonstrated noticeable results in the design of architectures for image segmentation and classification. Despite the considerable success of the architecture search in image segmentation and classification, it is still an unresolved and urgent problem. Moreover, the neural architecture search is also a highly computationally expensive task. This work proposes a new approach based on a genetic algorithm to search for the optimal convolutional neural network architecture. We integrated a genetic algorithm with standard stochastic gradient descent that implements weight distribution across all architecture solutions. This approach utilises a genetic algorithm to design a sub-graph of a convolution cell, which maximises the accuracy on the validation set. We show the performance of our approach on the CIFAR-10 and CIFAR-100 datasets with a final accuracy of 93.21% and 78.89%, respectively. The main scientific contribution of our work is the combination of genetic algorithm with weight distribution in the architecture search tasks that achieve similar to state-of-the-art results on a single GPU. Keywords: convolutional neural networks, genetic algorithms, weight distribution, ablation study. П.М. РАДЮК Хмельницький національний університет ЗАСТОСУВАННЯ ГЕНЕТИЧНОГО АЛГОРИТМУ ДЛЯ ПОШУКУ ОПТИМАЛЬНОЇ АРХІТЕКТУРИ ЗГОРТКОВОЇ НЕЙРОННОЇ МЕРЕЖІ З РОЗПОДІЛЕННЯМ ВАГ За останнє десятиліття новий спосіб дослідження нейронних мереж під назвою «Пошук мережевих архітектур» продемонстрував позитивні результати в розробці архітектур для сегментації та класифікації зображень. Незважаючи на значний успіх пошуку архітектур в задачах сегментації та класифікації зображень, він все ще є невирішеною і актуальною проблемою. Більше того, пошук архітектур нейронних мереж є також дуже витратим з точки зору обчислювальних ресурсів. У цій роботі пропонується новий підхід на основі генетичного алгоритму для пошуку оптимальної архітектури згорткової нейронної мережі. Ми інтегрували генетичний алгоритм зі стандартним стохастичним градієнтом, що реалізує розподіл ваг у всіх архітектурних рішеннях. Цей підхід використовує генетичний алгоритм для проектування частини графу в якості згорткового шару, що забезпечує максимальну точність на валідаційному наборі даних. У цій роботі ми демонструємо ефективність нашого підходу на наборах даних CIFAR-10 та CIFAR-100 з кінцевою точністю 93,21 % та 78,89 % відповідно. Основним науковим внеском нашої роботи є поєднання генетичного алгоритму з розподілом ваг в задачах пошуку архітектури, що досягає точності класифікацїі зображення з використанням одного графічного процесора близької до найсучасніших результатів. Ключові слова: згорткові нейронні мережі, генетичні алгоритми, розподілення ваг, абляція дослідження. Introduction In recent decades, artificial neural networks have produced outstanding results in computer vision with a wide variety of applied tasks such as object detection, image segmentation and classification and others. The design of neural network architectures for a specific task or dataset usually requires specific approaches and a large number of computational resources [1, 2]. Recently, a new way in neural networks research called Network Architectures Search (NAS) has demonstrated noticeable results in the design of architectures for image segmentation and classification. NAS approaches use a recurrent neural network (RNN) controller to generate a candidate network architecture, called child model, which is then trained to converge. After the training, researches measure the performance of the trained network architecture on the desired task or dataset. RNN controller receives the performance measurement as a signal to explore for a better architecture. After that, this process repeats over many computationally expensive iterations. Analysis of recent research Despite the considerable success of NAS in classification tasks, it is still highly computationally expensive. Recently, many studies have used the idea of parameter sharing [3] across all child models to reduce the need for training each child model from scratch, thereby eliminating most computational costs. Weight distribution, the analogue of parameter sharing for convolutional cells, has shown prominent result utilising reinforcement-based and gradient-based methods. The first method is an effective search for neural architecture using parameter sharing (ENAS) [4] based on reinforcement training with the RNN controller to create the candidate architecture. The second method is called a differentiable architecture search with various modifications (DARTS) [5], where each compound has a gradient-updating probability function. However, none of the publications has yet combined GAs with parameter sharing or its analogies to NAS. Genetic algorithm (GA) is a search method based on natural selection and genetics [6]. GAs consist of four fundamental concepts: selection, cross-over, mutation, and replacement. Population, another essential component of GA, is utilised to generate new candidate solutions. A new generation is created in each iteration, using a three-step process of selection, cross-over and mutation. The next generation is then inserted into the population through the replacement phase. The algorithm starts with a random set that is evaluated at the beginning of training.

With the advent of new technologies and advancement in medical science we are trying to process the information artificially as our biological system performs inside our body. Artificial intelligence through a biological word is realized based on mathematical equations and artificial neurons. Our main focus is on the implementation of Neural Network Architecture (NNA) with on a chip learning in analog VLSI for generic signal processing applications. In the proposed paper analog components like Gilbert Cell Multiplier (GCM), Neuron activation Function (NAF) are used to implement artificial NNA. The analog components used are comprises of multipliers and adders’ along with the tan-sigmoid function circuit using MOS transistor in subthreshold region. This neural architecture is trained using Back propagation (BP) algorithm in analog domain with new techniques of weight storage. Layout design and verification of the proposed design is carried out using Tanner EDA 14.1 tool and synopsys Tspice. The technology used in designing the layouts is MOSIS/HP 0.5u SCN3M, Tight Metal.

As animals adapt to their environments, their brains are tasked with processing stimuli in different sensory contexts. Whether these computations are context dependent or independent, they are all implemented in the same neural tissue. A crucial question is what neural architectures can respond flexibly to a range of stimulus conditions and switch between them. This is a particular case of flexible architecture that permits multiple related computations within a single circuit. Here, we address this question in the specific case of the visual system circuitry, focusing on context integration, defined as the integration of feedforward and surround information across visual space. We show that a biologically inspired microcircuit with multiple inhibitory cell types can switch between visual processing of the static context and the moving context. In our model, the VIP population acts as the switch and modulates the visual circuit through a disinhibitory motif. Moreover, the VIP population is efficient, requiring only a relatively small number of neurons to switch contexts. This circuit eliminates noise in videos by using appropriate lateral connections for contextual spatio-temporal surround modulation, having superior denoising performance compared to circuits where only one context is learned. Our findings shed light on a minimally complex architecture that is capable of switching between two naturalistic contexts using few switching units. Author Summary The brain processes information at all times and much of that information is context-dependent. The visual system presents an important example: processing is ongoing, but the context changes dramatically when an animal is still vs. running. How is context-dependent information processing achieved? We take inspiration from recent neurophysiology studies on the role of distinct cell types in primary visual cortex (V1).We find that relatively few "switching units"-akin to the VIP neuron type in V1 in that they turn on and off in the running vs. still context and have connections to and from the main population-is sufficient to drive context dependent image processing. We demonstrate this in a model of feature integration, and in a test of image denoising. The underlying circuit architecture illustrates a concrete computational role for the multiple cell types under increasing study across the brain, and may inspire more flexible neurally inspired computing architectures.

This piece of research introduces a purely data-driven, directly reconfigurable, divide-and-conquer on-line monitoring (OLM) methodology for automatically selecting the minimum number of neutron detectors (NDs)-and corresponding neutron noise signals (NSs)-which are currently necessary, as well as sufficient, for inspecting the entire nuclear reactor (NR) in-core area. The proposed implementation builds upon the 3-tuple configuration, according to which three sufficiently pairwise-correlated NSs are capable of on-line (I) verifying each NS of the 3-tuple and (II) endorsing correct functioning of each corresponding ND, implemented herein via straightforward pairwise comparisons of fixed-length sliding time-windows (STWs) between the three NSs of the 3-tuple. A pressurized water NR (PWR) model-developed for H2020 CORTEX-is used for deriving the optimal ND/NS configuration, where (i) the evident partitioning of the 36 NDs/NSs into six clusters of six NDs/NSs each, and (ii) the high cross-correlations (CCs) within every 3-tuple of NSs, endorse the use of a constant pair comprising the two most highly CC-ed NSs per cluster as the first two members of the 3-tuple, with the third member being each remaining NS of the cluster, in turn, thereby computationally streamlining OLM without compromising the identification of either deviating NSs or malfunctioning NDs. Tests on the in-core dataset of the PWR model demonstrate the potential of the proposed methodology in terms of suitability for, efficiency at, as well as robustness in ND/NS selection, further establishing the "directly reconfigurable" property of the proposed approach at every point in time while using one-third only of the original NDs/NSs.