Neural Network Architecture

This paper aims to present a comprehensive review of advanced techniques and models with a specific focus on deep neural network (DNN) for resource-constrained environments (RCE). The paper contributes by highlighting the RCE devices, analyzing challenges, reviewing a broad range of optimization techniques and DNN models, and offering a comparative assessment. The findings provide potential optimization techniques and recommend a baseline model for future development. It encompasses a broad range of DNN optimization techniques, including network pruning, weight quantization, knowledge distillation, depthwise separable convolution, residual connections, factorization, dense connections, and compound scaling. Moreover, the review analyzes the established optimization models which utilizes the above optimization techniques. A comprehensive analysis is conducted for each technique and model, considering its specific attributes, usability, strengths, and limitations in the context of effective deployment in RCEs. The review also presents a comparative assessment of advanced DNN models' deployment for image classification, employing key evaluation metrics such as accuracy and efficiency factors like memory and inference time. The article concludes with the finding that combining depthwise separable convolution, weight quantization, and pruning represents potential optimization techniques, while also recommending EfficientNetB1 as a baseline model for the future development of optimization models in RCE image classification.

Air surveillance radar tracking systems present a variety of known problems related to uncertainty and lack of accurately in radar measurements used as source in these systems. In this work, we feature the theoretical aspects of a tracking algorithm based on neural network paradigm where, from discrete measurements provided by surveillance radar, the objective will be to estimate the target state for tracking purposes as accuracy as possible. The absence of an optimal statistical solution makes the featured neural network attractive despite the availability of complex and well-known filtering algorithms. Neural networks exhibit universal mapping capabilities that allow them to be used as a control tool for capturing hidden information about models learned from a dataset. We use these capabilities to let the network learn, not only from the received radar measurement information, but also from the aircraft maneuvering context, contextual information, where tracking application is wor...

This paper aims to investigate whether applying the Singular Value Decomposition (SVD) technique can reduce the workload of Convolutional Neural Networks (CNNs) without compromising image classification results. Usual methods for reducing file size are generally unsuitable for deployment because they are costly to train, not straightforward to use, or require additional resources. We performed a low-rank approximation of the CNN weights using the singular value decomposition (SVD) approach. Results for ResNet-50 were obtained from CIFAR-10, and EfficientNet-B0 was assessed using ImageNet. The assessment used metrics including accuracy, precision, recall, and F1-score. Using SVD at a 0.5 compression level, we reduced the network size of ResNet-50 on CIFAR-10 by 41.7% without significantly compromising accuracy (95.4%). With aggressive compression at a ratio of 0.1, the model achieved an 88.9% reduction in parameters, but accuracy decreased to 72.9%. Furthermore, the model increased inference speed slightly (up to 1.0x) and maintained an approximately 89.96 MB size. The uncompressed model was 10.4% accurate before training. Rather than other techniques, SVD offers straightforward yet practical suggestions for enhancing CNN performance on small datasets, without significantly compromising accuracy. These results demonstrate that SVD-based compression is a promising approach for utilizing CNNs in limitedresource systems. It achieves nearly complete accuracy with fewer parameters and has a minimal impact on the speed of CNNs.

The scaling hypothesis, the long-standing paradigm that posited predictable performance gains in artificial intelligence from increasing model size, data, and compute, is encountering significant and compounding limitations. Empirical evidence reveals a landscape of diminishing returns, where exponential resource investments yield only marginal capability improvements, challenging the economic and scientific viability of a scale-centric approach. This has catalyzed a fundamental paradigm shift in AI research, pivoting the field from a philosophy of "bigger is better" to one of "smarter is better." This paper provides a comprehensive synthesis and analysis of this critical inflection point. It begins by detailing the economic and scientific pressures-from unsustainable training costs and energy demands to performance plateaus on key benchmarks-that are forcing this re-evaluation. Subsequently, it surveys the multifaceted research vectors that define the new frontier of efficiency, including algorithmic optimizations such as quantization and knowledge distillation, architectural innovations like sparse Mixture-of-Experts (MoE), and the co-evolution of hardware and software. The paper further argues that inherent model limitations, such as hallucination and deficient multi-step reasoning, are being productively reframed from mere bugs into foundational research challenges, driving a deeper inquiry into the mechanisms of artificial cognition. Finally, it explores the forward-looking pathways to more robust and capable intelligence, including Retrieval-Augmented Generation (RAG) for verifiable knowledge grounding, model self-improvement via synthetic data, cognitive alignment through neuroscientific inspiration, and the integration of neurosymbolic methods for formal reasoning. This work documents a field in transition, moving beyond brute-force computation toward a more principled, efficient, and sustainable quest for artificial intelligence.

Convolutional neural networks (CNNs) have shown remarkable performance in various real-world applications. Unfortunately, the promising performance of CNNs can be achieved only when their architectures are optimally constructed. The architectures of state-of-the-art CNNs are typically hand-crafted with extensive expertise in both CNNs and the investigated data, which consequently hampers the widespread adoption of CNNs for less experienced users. Evolutionary deep learning (EDL) is able to automatically design the best CNN architectures without much expertise. However, existing EDL algorithms generally evaluate the fitness of a new architecture by training from scratch, resulting in the prohibitive computational cost even operated on high-performance computers. In this paper, an endto-end offline performance predictor based on the random forest is proposed to accelerate the fitness evaluation in EDL. The proposed performance predictor shows promising performance in term of the classification accuracy and the consumed computational resources when compared with 18 state-of-the-art peer competitors by integrating it into an existing EDL algorithm as a case study. The proposed performance predictor is also compared with the other two representatives of existing performance predictors. The experimental results show the proposed performance predictor not only significantly speeds up the fitness evaluations, but also achieves the best prediction among the peer performance predictors.

The performance of Convolutional Neural Networks (CNNs) highly relies on their architectures. In order to design a CNN with promising performance, extensive expertise in both CNNs and the investigated problem domain is required, which is not necessarily available to every interested user. To address this problem, we propose to automatically evolve CNN architectures by using a genetic algorithm based on ResNet and DenseNet blocks. The proposed algorithm is completely automatic in designing CNN architectures. In particular, neither pre-processing before it starts nor post-processing in terms of CNNs is needed. Furthermore, the proposed algorithm does not require users with domain knowledge on CNNs, the investigated problem or even genetic algorithms. The proposed algorithm is evaluated on the CIFAR10 and CIFAR100 benchmark datasets against 18 state-of-the-art peer competitors. Experimental results show that the proposed algorithm outperforms state-of-the-art CNNs hand-crafted and CNNs designed by automatic peer competitors in terms of the classification performance, and achieves a competitive classification accuracy against semi-automatic peer competitors. In addition, the proposed algorithm consumes much less computational resource than most peer competitors in finding the best CNN architectures.

We propose a novel convolutional neural networks (CNNs) training procedure to allow dynamically trade-offs between different resource and performance requirements. Our approach prioritizes the channels to enable structured sparsity and multi-fidelity approximations at inference time. We train the VGG network with our method on various benchmark datasets. The experiment results show that, on the CIFAR-10 dataset, a 63× parameters reduction and a 11× FLOPs reduction can be achieved, with only a 2% accuracy drop.

We propose a novel convolutional neural networks (CNNs) training procedure to allow dynamically trade-offs between different resource and performance requirements. Our approach prioritizes the channels to enable structured sparsity and multi-fidelity approximations at inference time. We train the VGG network with our method on various benchmark datasets. The experiment results show that, on the CIFAR-10 dataset, a 63× parameters reduction and a 11× FLOPs reduction can be achieved, with only a 2% accuracy drop.

Tiny deep learning has attracted increasing attention driven by the substantial demand for deploying deep learning on numerous intelligent Internet-of-Things devices. However, it is still challenging to unleash tiny deep learning's full potential on both large-scale datasets and downstream tasks due to the under-fitting issues caused by the limited model capacity of tiny neural networks (TNNs). To this end, we propose a framework called NetBooster to empower tiny deep learning by augmenting the architectures of TNNs via an expansion-then-contraction strategy. Extensive experiments show that NetBooster consistently outperforms state-of-the-art tiny deep learning solutions.

Artificial neural network play an important role in VLSI circuit to find and diagnosis multiple fault in digital circuit. In this paper, the example of single layer and multilayer neural network had been discussed secondly implement those structure by using verilog code and same idea must be implement in mat lab for getting number of iteration and verilog code gives us time taken to adjust the weight when error become almost equal to zero. The purposed aim at reducing resource requirement, without much compromises on the speed that neural network can be realized on single chip at lower cost.

This study explores the integration of neural network emulators into Computer-Aided Design (CAD) environments, aiming to enhance neural architecture modeling. The objective is to merge the computational power of neural networks with the precision and control offered by CAD systems. By embedding neural network emulators, the research seeks to enable the simulation, modeling, and refinement of complex neural architectures within a CAD framework. This integration allows for a more interactive and dynamic environment, facilitating the exploration and optimization of various neural structures. The study anticipates that this approach will lead to significant advancements in the design of efficient and versatile neural networks, with applications ranging from automated learning systems to advanced problem-solving mechanisms. Ultimately, this research aims to expand the boundaries of computational design by leveraging neural network emulation within CAD environments, opening up new possibilities in the development of innovative neural architectures.

Diabetic retinopathy (DR) is a major cause of vision loss globally, making early detection vital for effective intervention. However, manual screening is prone to errors and inefficiency. Automated solutions using deep learning models have shown promise, yet they face challenges in balancing accuracy and computational efficiency, particularly in resource-constrained environments. The main objective of this research is to optimize the MobileNetV2 deep learning model for detecting diabetic retinopathy, filling the gap in achieving both high accuracy and low computational demands. The proposed method introduces novel optimization techniques, including model pruning, to reduce model complexity without sacrificing accuracy. These optimizations target the deployment of the model on mobile and edge devices, ensuring real-time performance. The optimized MobileNetV2 model was evaluated on a DR dataset, showing improved efficiency with a 50% reduction in model weights, leading to faster inference times while maintaining an accuracy of 95.91%. The results confirm that the pruned model performs on par with the original in terms of accuracy but with enhanced suitability for real-time applications.

This research paper presents an advanced exploration of the Φπε framework, an integrative model that unifies quantum mechanics, consciousness studies, and metaphysical principles under the auspices of harmony (Φ), quantum entanglement (ε), and transcendence (π). The framework is examined through its mathematical foundations, symbolic language, and practical applications, providing a comprehensive model that transcends disciplinary boundaries. The paper further proposes novel equations and conceptual models that enhance the utility of Φπε in both theoretical and practical contexts, particularly within quantum computing and consciousness studies. The potential of Φπε as a universal language and its implications for future scientific and philosophical inquiries are also discussed.

We consider network sparsification as an L0-norm regularized binary optimization problem, where each unit of a neural network (e.g., weight, neuron, or channel, etc.) is attached with a stochastic binary gate, whose parameters are jointly optimized with original network parameters. The Augment-Reinforce-Merge (ARM) [28], a recently proposed unbiased gradient estimator, is investigated for this binary optimization problem. Compared to the hard concrete gradient estimator from Louizos et al. [20], ARM demonstrates superior performance of pruning network architectures while retaining almost the same accuracies of baseline methods. Similar to the hard concrete estimator, ARM also enables conditional computation during model training but with improved effectiveness due to the exact binary stochasticity. Thanks to the flexibility of ARM, many smooth or non-smooth parametric functions, such as scaled sigmoid or hard sigmoid, can be used to parameterize this binary optimization problem and the unbiasness of the ARM estimator is retained, while the hard concrete estimator has to rely on the hard sigmoid function to achieve conditional computation and thus accelerated training. Extensive experiments on multiple public datasets demonstrate state-ofthe-art pruning rates with almost the same accuracies of baseline methods. The resulting algorithm L0-ARM sparsifies the Wide-ResNet models on CIFAR-10 and CIFAR-100 while the hard concrete estimator cannot. The code is public available at https://github.com/leo-yangli/l0-arm.

Model compression is an essential technique for deploying deep neural networks (DNNs) on power and memory-constrained resources. However, existing model-compression methods often rely on human expertise and focus on parameters' local importance, ignoring the rich topology information within DNNs. In this paper, we propose a novel multi-stage graph embedding technique based on graph neural networks (GNNs) to identify DNN topologies and use reinforcement learning (RL) to find a suitable compression policy. We performed resource-constrained (i.e., FLOPs) channel pruning and compared our approach with state-of-the-art model compression methods. We evaluated our method on various models from typical to mobile-friendly networks, such as ResNet family, VGG-16, MobileNet-v1/v2, and Shuf-fleNet. Results show that our method can achieve higher compression ratios with a minimal finetuning cost yet yields outstanding and competitive performance. The code is open-sourced at https://github.com/yusx-swapp/ GNN-RL-Model-Compression.

Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network quantization (SLQ) for high-bit quantization and multi-level network quantization (MLQ) for extremely low-bit quantization (ternary). We are the first to consider the network quantization from both width and depth level. In the width level, parameters are divided into two parts: one for quantization and the other for re-training to eliminate the quantization loss. SLQ leverages the distribution of the parameters to improve the width level. In the depth level, we introduce incremental layer compensation to quantize layers iteratively which decreases the quantization loss in each iteration. The proposed approaches are validated with extensive experiments based on the state...

Recent research has focused on weight sparsity in neural network training to reduce FLOPs, aiming for improved efficiency (test accuracy w.r.t training FLOPs). However, sparse weight training often sacrifices accuracy, requiring extended training schedules to attain the accuracy of dense models. In contrast, our approach, Sparse Iso-FLOP Transformations (Sparse-IFT), uses sparsity to improve accuracy while maintaining dense model FLOPs. Using a single hyperparameter (i.e., sparsity level), Sparse-IFTs efficiently replace dense layers, expanding the search space for optimal sparse masks. In addition, dynamic sparse training with Sparse-IFT models effectively navigates this larger sparse mask-weight space, which is evidenced by a spectral analysis using Ramanujan graph properties. Our study reveals a robust correlation among mask topology, weights, and final performance. Notably, without adjusting hyperparameters, replacing dense layers with Sparse-IFT yields significant improvements, such as a +3.5% boost for ResNet-18 on ImageNet and +0.9% for GPT-3 Small on the Open LLM leaderboard. To our knowledge, this is the first work to demonstrate the use of sparsity for improving the accuracy of dense models through a simple-to-use set of sparse transformations.

In this paper, a new pruning strategy based on the neuroplasticity of biological neural networks is presented. The novel pruning algorithm proposed is inspired by the knowledge remapping ability after injuries in the cerebral cortex. Thus, it is proposed to simulate induced injuries into the network by pruning full convolutional layers or entire blocks, assuming that the knowledge from the removed segments of the network may be remapped and compressed during the recovery (retraining) process. To reconnect the remaining segments of the network, a translator block is introduced. The translator is composed of a pooling layer and a convolutional layer. The pooling layer is optional and placed to ensure that the spatial dimension of the feature maps matches across the pruned segments. After that, a convolutional layer (simulating the intact cortex) is placed to ensure that the depth of the feature maps matches and is used to remap the removed knowledge. As a result, lightweight, efficien...

Network Morphism based Neural Architecture Search (NAS) is one of the most efficient methods, however, knowing where and when to add new neurons or remove dis-functional ones is generally left to black-box Reinforcement Learning models. In this paper, we present a new Network Morphism based NAS called Noisy Heuristics NAS which uses heuristics learned from manually developing neural network models and inspired by biological neuronal dynamics. Firstly, we add new neurons randomly and prune away some to select only the best fitting neurons. Secondly, we control the number of layers in the network using the relationship of hidden units to the number of input-output connections. Our method can increase or decrease the capacity or non-linearity of models online which is specified with a few meta-parameters by the user. Our method generalizes both on toy datasets and on real-world data sets such as MNIST, CIFAR-10, and CIFAR-100. The performance is comparable to the handengineered architecture ResNet-18 with the similar parameters.

Recent research has focused on weight sparsity in neural network training to reduce FLOPs, aiming for improved efficiency (test accuracy w.r.t training FLOPs). However, sparse weight training often sacrifices accuracy, requiring extended training schedules to attain the accuracy of dense models. In contrast, our approach, Sparse Iso-FLOP Transformations (Sparse-IFT), uses sparsity to improve accuracy while maintaining dense model FLOPs. Using a single hyperparameter (i.e., sparsity level), Sparse-IFTs efficiently replace dense layers, expanding the search space for optimal sparse masks. In addition, dynamic sparse training with Sparse-IFT models effectively navigates this larger sparse mask-weight space, which is evidenced by a spectral analysis using Ramanujan graph properties. Our study reveals a robust correlation among mask topology, weights, and final performance. Notably, without adjusting hyperparameters, replacing dense layers with Sparse-IFT yields significant improvements, such as a +3.5% boost for ResNet-18 on ImageNet and +0.9% for GPT-3 Small on the Open LLM leaderboard. To our knowledge, this is the first work to demonstrate the use of sparsity for improving the accuracy of dense models through a simple-to-use set of sparse transformations.

Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, calle...

Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Most existing NAS approaches require one complete search for each deployment specification of hardware or objective. This is a computationally impractical endeavor given the potentially large number of application scenarios. In this paper, we propose Neural Architecture Transfer (NAT) to overcome this limitation. NAT is designed to efficiently generate task-specific custom models that are competitive even under multiple conflicting objectives. To realize this goal we learn task-specific supernets from which specialized subnets can be sampled without any additional training. The key to our approach is an integrated online transfer learning and many-objective evolutionary search procedure. A pre-trained supernet is iteratively adapted while simultaneously searching for task-specific subnets. We demonstrate the efficacy of NAT on 11 benchmark image classification tasks ranging from large-scale multi-class to small-scale fine-grained datasets. In all cases, including ImageNet, NATNets improve upon the state-of-the-art under mobile settings (≤ 600M Multiply-Adds). Surprisingly, small-scale fine-grained datasets benefit the most from NAT. At the same time, the architecture search and transfer is orders of magnitude more efficient than existing NAS methods. Overall, experimental evaluation indicates that, across diverse image classification tasks and computational objectives, NAT is an appreciably more effective alternative to fine-tuning based transfer learning. Code is available at https://github.com/human-analysis/neural-architecture-transfer.

In recent developments, the significance of Edge AI has come to the forefront. Edge devices, which encompass a wide array of IoT devices and embedded systems, benefit from the deployment of efficient and compact neural network models. Edge AI faces a significant hurdle due to their resource-constrained nature in terms of computation and memory. Deploying these models on resource-constrained devices like edge devices becomes challenging. To address this issue, quantization has emerged as an efficient strategy to reduce the computational and memory requirements, enabling their deployment on edge devices. Yet, a pivotal concern lies in preserving the performance of quantized models at levels comparable to their original floating-point counterparts. In this paper, we review the details of quantization, including its foundational assumptions, techniques and tradeoffs that influence the quantization accuracy to achieve optimal results while embracing quantization's benefits.

Most deep architectures for image classification-even those that are trained to classify a large number of diverse categories-learn shared image representations with a single model. Intuitively, however, categories that are more similar should share more information than those that are very different. While hierarchical deep networks address this problem by learning separate features for subsets of related categories, current implementations require simplified models using fixed architectures specified via heuristic clustering methods. Instead, we propose Blockout, a method for regularization and model selection that simultaneously learns both the model architecture and parameters. A generalization of Dropout, our approach gives a novel parametrization of hierarchical architectures that allows for structure learning via back-propagation. To demonstrate its utility, we evaluate Blockout on the CIFAR and ImageNet datasets, demonstrating improved classification accuracy, better regularization performance, faster training, and the clear emergence of hierarchical network structures.

Convolutional neural networks (CNNs) introduce state-ofthe-art results for various tasks with the price of high computational demands. Inspired by the observation that spatial correlation exists in CNN output feature maps (ofms), we propose a method to dynamically predict whether ofm activations are zero-valued or not according to their neighboring activation values, thereby avoiding zero-valued activations and reducing the number of convolution operations. We implement the zero activation predictor (ZAP) with a lightweight CNN, which imposes negligible overheads and is easy to deploy on existing models. ZAPs are trained by mimicking hidden layer ouputs; thereby, enabling a parallel and label-free training. Furthermore, without retraining, each ZAP can be tuned to a different operating point trading accuracy for MAC reduction.

The performance of Convolutional Neural Networks (CNNs) highly relies on their architectures. In order to design a CNN with promising performance, extensive expertise in both CNNs and the investigated problem domain is required, which is not necessarily available to every interested user. To address this problem, we propose to automatically evolve CNN architectures by using a genetic algorithm based on ResNet and DenseNet blocks. The proposed algorithm is completely automatic in designing CNN architectures. In particular, neither pre-processing before it starts nor post-processing in terms of CNNs is needed. Furthermore, the proposed algorithm does not require users with domain knowledge on CNNs, the investigated problem or even genetic algorithms. The proposed algorithm is evaluated on the CIFAR10 and CIFAR100 benchmark datasets against 18 state-of-the-art peer competitors. Experimental results show that the proposed algorithm outperforms state-of-the-art CNNs hand-crafted and CNNs designed by automatic peer competitors in terms of the classification performance, and achieves a competitive classification accuracy against semi-automatic peer competitors. In addition, the proposed algorithm consumes much less computational resource than most peer competitors in finding the best CNN architectures.

Animals display rich and coordinated motor patterns during walking and running. Previous modelling as well as experimental results suggest that the balance between excitation and inhibition in neural networks may be critical for generating such structured motor patterns. However, biological neural networks have an anatomical imbalance between excitatory and inhibitory neural populations. We explore the influence of such an anatomical imbalance on the ability of a reservoir computing artificial neural network to learn human locomotor patterns for slow walking, fast walking and running. We varied the numbers of neurons, connections percentages and connection strengths of excitatory and inhibitory populations. We showed that performance depended on the network anatomy. First, it deteriorated when the total number of neurons was too small or the total connection strength was too large. Second, performance was critically dependent on the balance between excitation and inhibition. Imbalan...

Although radar and communications signal classification are usually treated separately, they share similar characteristics, and methods applied in one domain can be potentially applied in the other. We propose a simple and unified scheme for the classification of radar and communications signals using Long Short-Term Memory (LSTM) neural networks. This proposal provides an improvement of the state of the art on radar signals where LSTM models are starting to be applied within schemes of higher complexity. To date, there is no standard public dataset for radar signals. Therefore, we propose DeepRadar2022 1 , a radar dataset used in our systematic evaluations that is available publicly and will facilitate a standard comparison between methods.

State-of-the-art computer vision models are rapidly increasing in capacity, where the number of parameters far exceeds the number required to fit the training set. This results in better optimization and generalization performance. However, the huge size of contemporary models results in large inference costs and limits their use on resource-limited devices. In order to reduce inference costs, convolutional filters in trained neural networks could be pruned to reduce the run-time memory and computational requirements during inference. However, severe post-training pruning results in degraded performance if the training algorithm results in dense weight vectors. We propose the use of Batch Bridgeout, a sparsity inducing stochastic regularization scheme, to train neural networks so that they could be pruned efficiently with minimal degradation in performance. We evaluate the proposed method on common computer vision models VGGNet, ResNet and Wide-ResNet on the CIFAR10 and CIFAR100 image classification tasks. For all the networks, experimental results show that Batch Bridgeout trained networks achieve higher accuracy across a wide range of pruning intensities compared to Dropout and weight decay regularization. INDEX TERMS Bridge regularization, deep neural networks, dropout, pruning, image classification, neural network regularization, neural network training.

When deploying deep learning models to a device, it is traditionally assumed that available computational resources (compute, memory, and power) remain static. However, real-world computing systems do not always provide stable resource guarantees. Computational resources need to be conserved when load from other processes is high or battery power is low. Inspired by recent works on neural network subspaces, we propose a method for training a compressible subspace of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. Our models require no retraining, thus our subspace of models can be deployed entirely on-device to allow adaptive network compression at inference time. We present results for achieving arbitrarily finegrained accuracy-efficiency trade-offs at inference time for structured and unstructured sparsity. We achieve accuracies on-par with standard models when testing our uncompressed models, and maintain high a...

Previous neural architecture search (NAS) approaches for mobile platforms have achieved great success in designing a slim-but-accurate neural network that is generally wellmatched to a single computing unit such as a CPU or GPU. However, as recent mobile devices consist of multiple heterogeneous computing units, the next main challenge is to maximize both accuracy and efficiency by fully utilizing multiple available resources. We propose an ensemble-like approach with intermediate feature aggregations, namely synchronizations, for active collaboration between individual models on a mobile device. A main challenge is to determine the optimal synchronization strategies for achieving both performance and efficiency. To this end, we propose SyncNAS to automate the exploration of synchronization strategies for collaborative neural architectures that maximize utilization of heterogeneous computing units on a target device. We introduce a novel search space for synchronization strategy and apply Monte Carlo tree search (MCTS) algorithm to improve the sampling efficiency and reduce the search cost. On ImageNet, our collaborative model based on Mo-bileNetV2 achieves 2.7% top-1 accuracy improvement within the baseline latency budget. Under the reduced target latency down to half, our model maintains higher accuracy than its * Current affiliation: OctoML

Humans and other animals navigate different landscapes and environments with ease, a feat that requires the brain’s ability to rapidly and accurately adapt to different visual domains, generalizing across contexts/backgrounds. Despite recent progress in deep learning applied to classification and detection in the presence of multiple confounds including contextual ones [25, 30], there remain important challenges to address regarding how networks can perform context-dependent computations and how contextually-invariant visual concepts are formed. For instance, recent studies have shown artificial networks that repeatedly misclassified familiar objects set on new backgrounds, e.g. incorrectly labelling known animals when they appeared in a different setting [3]. Here, we show how a bio-inspired network motif can explicitly address this issue. We do this using a novel dataset which can be used as a benchmark for future studies probing invariance to backgrounds. The dataset consists of ...

The detection of novel stimuli is critical to learn and survive in a dynamic environment. Though novel stimuli powerfully affect brain activity, their impact on specific cell types and circuits is not well understood. Disinhibition is one candidate mechanism for novelty-induced enhancements in activity. Here we characterize the impact of stimulus novelty on disinhibitory circuit components using longitudinal 2-photon calcium imaging of Vip, Sst, and excitatory populations in the mouse visual cortex. Mice learn a behavioral task with stimuli that become highly familiar, then are tested on both familiar and novel stimuli. Mice consistently perform the task with novel stimuli, yet responses to stimulus presentations and stimulus omissions are dramatically altered. Further, we find that novelty modifies coding of visual as well as behavioral and task information. At the population level, the direction of these changes is consistent with engagement of the Vip-Sst disinhibitory circuit. A...

Implicit Neural Representation (INR) is an innovative approach for representing complex shapes or objects without explicitly defining their geometry or surface structure. Instead, INR represents objects as continuous functions. Previous research has demonstrated the effectiveness of using neural networks as INR for image compression, showcasing comparable performance to traditional methods such as JPEG. However, INR holds potential for various applications beyond image compression. This paper introduces Rapid-INR, a novel approach that utilizes INR for encoding and compressing images, thereby accelerating neural network training in computer vision tasks. Our methodology involves storing the whole dataset directly in INR format on a GPU, mitigating the significant data communication overhead between the CPU and GPU during training. Additionally, the decoding process from INR to RGB format is highly parallelized and executed on-the-fly. To further enhance compression, we propose iterative and dynamic pruning, as well as layer-wise quantization, building upon previous work. We evaluate our framework on the image classification task, utilizing the ResNet-18 backbone network and three commonly used datasets with varying image sizes. Rapid-INR reduces memory consumption to only 5% of the original dataset size and achieves a maximum 6× speedup over the PyTorch training pipeline, as well as a maximum 1.2× speedup over the DALI training pipeline, with only a marginal decrease in accuracy. Importantly, Rapid-INR can be readily applied to other computer vision tasks and backbone networks with reasonable engineering efforts. Our implementation code is publicly available at https://github.com/sharc-lab/Rapid-INR.

Efficient continual learning in humans is enabled by a rich set of neurophysiological mechanisms and interactions between multiple memory systems. The brain efficiently encodes information in non-overlapping sparse codes, which facilitates the learning of new associations faster with controlled interference with previous associations. To mimic sparse coding in DNNs, we enforce activation sparsity along with a dropout mechanism which encourages the model to activate similar units for semantically similar inputs and have less overlap with activation patterns of semantically dissimilar inputs. This provides us with an efficient mechanism for balancing the reusability and interference of features, depending on the similarity of classes across tasks. Furthermore, we employ sparse coding in a multiple-memory replay mechanism. Our method maintains an additional long-term semantic memory that aggregates and consolidates information encoded in the synaptic weights of the working model. Our extensive evaluation and characteristics analysis show that equipped with these biologically inspired mechanisms, the model can further mitigate forgetting 1 .

In this paper, we present the implementation of a system that identifies the modulation of complex radio signals. This is realized using an artificial intelligence model developed, trained, and integrated with Microsoft Azure cloud. We consider that cloud-based platforms offer enough flexibility and processing power to use them instead of conventional computers for signal processing based on artificial intelligence. We tested the implementation using a software-defined radio platform developed in GNU Radio that generates and receives real modulated signals. This process ensures that the solution proposed is viable to be used in real signal processing systems. The results obtained show that for certain modulation types, the identification is performed with a high degree of success. The use of a cloud-based platform allows quick access to the system. The user is able to identify the signal modulation using only a laptop that has access to the internet.

The majority of synaptic inputs to the primary visual cortex (V1) are non-feedforward, instead originating from local and anatomical feedback connections. Animal electrophysiology experiments show that feedback signals originating from higher visual areas with larger receptive fields modulate the surround receptive fields of V1 neurons. Theories of cortical processing propose various roles for feedback and feedforward processing, but systematically investigating their independent contributions to cortical processing is challenging because feedback and feedforward processes coexist even in single neurons. Capitalising on the larger receptive fields of higher visual areas compared to primary visual cortex (V1), we used an occlusion paradigm that isolates top-down influences from feedforward processing. We utilised functional magnetic resonance imaging (fMRI) and multi-voxel pattern analysis methods in humans viewing natural scene images. We parametrically measured how the availability of contextual information determines the presence of detectable feedback information in non-stimulated V1, and how feedback information interacts with feedforward processing. We show that increasing the visibility of the contextual surround increases scene-specific feedback information, and that this contextual feedback enhances feedforward information. Our findings are in line with theories that cortical feedback signals transmit internal models of predicted inputs. .

The majority of synaptic inputs to the primary visual cortex (V1) are non-feedforward, instead originating from local and anatomical feedback connections. Animal electrophysiology experiments show that feedback signals originating from higher visual areas with larger receptive fields modulate the surround receptive fields of V1 neurons. Theories of cortical processing propose various roles for feedback and feedforward processing, but systematically investigating their independent contributions to cortical processing is challenging because feedback and feedforward processes coexist even in single neurons. Capitalising on the larger receptive fields of higher visual areas compared to primary visual cortex (V1), we used an occlusion paradigm that isolates top-down influences from feedforward processing. We utilised functional magnetic resonance imaging (fMRI) and multi-voxel pattern analysis methods in humans viewing natural scene images. We parametrically measured how the availability...

This paper aims to find an automatic solution for the modulation’s classification of different types of radio signals by relying on Artificial Intelligence. This project is part of a long process of Communications Intelligence looking for an automatic solution to demodulate, decode and decipher communication signals. Our work therefore consisted in the choice of the database needed for supervised deep learning, the evaluation of existing techniques on raw communication signals, and the proposal of a solution based on deep learning networks allowing to classify the types of modulation with an optimal ratio (computation time / accuracy).
We first carried out a research work on the existing models of automatic classification in order
to use them as a reference. We consequently proposed an ensemble learning approach based on
tuned ResNet and Transformer Neural Network that is efficient at extracting multi- scale
features from the raw I/Q sequence data and also considers the challenge of predicting in low
Signal Noise Ratio (SNR) conditions. In the end, we delivered an architecture that is easy to
handle and apply to communication signals. This solution has an optimal and robust
architecture that automatically determines the type of modulation with an accuracy up to 95%.

Keyword Spotting (KWS) has been the subject of research in recent years given the increase of embedded systems for command recognition such as Alexa, Google Home, and Siri. Performance, model size, processing time, and robustness to noise are fundamental in these systems. Furthermore, applications in embedded systems demand computationally efficient models that can be implemented in current technology. In this work, an approach for keyword recognition is evaluated using three deep learning models namely LeNet-5, SqueezeNet, and EfficientNet-B0. We evaluate transfer learning, pruning and quantization strategies in training and test using noisy and clean speech signals. In addition, compression techniques such as pruning and quantization were assessed in terms of the size reduction of the model footprint and the accuracy obtained in each case. Using the Google's Speech Commands dataset and additive babble noise signal, our keyword recognition approach achieves an accuracy of 94.6% using an unstructured pruning of 80% of the parameters of the original SqueezeNet network with a reduction of 70% in the model size. INDEX TERMS Speech recognition, machine learning algorithms, speech analysis, spectral analysis, pruning, quantization, keyword spotting. I. INTRODUCTION Voice commands are becoming a natural way to interact with consumer electronic devices [1], [2]. Systems with speech command recognition such as Amazon's Alexa, Apple's Siri, and Google's Assistant are examples of this popularity. These smart devices often use some embedded system (e.g., microcontrollers [3], microprocessors, field-programmable gate arrays, or dedicated devices [4]) with limited resources, making the implementation of speech recognition algorithms dependent on hardware limitations [5]. Typically, microcontrollers, the cheapest approach, have small memory capacity (i.e., a few kilobytes) and require energy-saving strategies since, in most cases, these edge devices are always active and they are generally powered by batteries. Additionally, they must have low latency and The associate editor coordinating the review of this manuscript and approving it for publication was Mounim A. El Yacoubi .

Spatial attention enhances sensory processing of goal-relevant information and improves perceptual sensitivity. Yet, the specific neural mechanisms underlying the effects of spatial attention on performance are still contested. Here, we examine different attention mechanisms in spiking deep convolutional neural networks. We directly contrast effects of precision (internal noise suppression) and two different gain modulation mechanisms on performance on a visual search task with complex real-world images. Unlike standard artificial neurons, biological neurons have saturating activation functions, permitting implementation of attentional gain as gain on a neuron’s input or on its outgoing connection. We show that modulating the connection is most effective in selectively enhancing information processing by redistributing spiking activity, and by introducing additional task-relevant information, as shown by representational similarity analyses. Precision only produced minor attentional...

In Convolutional Neural Networks (CNNs) information flows across a small neighbourhood of each pixel of an image, preventing long-range integration of features before reaching deep layers in the network. We propose a novel architecture that allows flexible information flow between features $z$ and locations $(x,y)$ across the entire image with a small number of layers. This architecture uses a cycle of three orthogonal convolutions, not only in $(x,y)$ coordinates, but also in $(x,z)$ and $(y,z)$ coordinates. We stack a sequence of such cycles to obtain our deep network, named CycleNet. As this only requires a permutation of the axes of a standard convolution, its performance can be directly compared to a CNN. Our model obtains competitive results at image classification on CIFAR-10 and ImageNet datasets, when compared to CNNs of similar size. We hypothesise that long-range integration favours recognition of objects by shape rather than texture, and we show that CycleNet transfers b...

Turning the weights to zero when training a neural network helps in reducing the computational complexity at inference. To progressively increase the sparsity ratio in the network without causing sharp weight discontinuities during training, our work combines soft-thresholding and straight-through gradient estimation to update the raw, i.e. non-thresholded, version of zeroed weights. Our method, named ST-3 for straight-through/soft-thresholding/sparsetraining 2 , obtains SoA results, both in terms of accuracy/sparsity and accuracy/FLOPS trade-offs, when progressively increasing the sparsity ratio in a single training cycle. In particular, despite its simplicity, ST-3 favorably compares to the most recent methods, adopting differentiable formulations [42] or bio-inspired neuroregeneration principles [25]. This suggests that the key ingredients for effective sparsification primarily lie in the ability to give the weights the freedom to evolve smoothly across the zero state while progressively increasing the sparsity ratio.

Turning the weights to zero when training a neural network helps in reducing the computational complexity at inference. To progressively increase the sparsity ratio in the network without causing sharp weight discontinuities during training, our work combines soft-thresholding and straight-through gradient estimation to update the raw, i.e. non-thresholded, version of zeroed weights. Our method, named ST-3 for straight-through/soft-thresholding/sparsetraining 2 , obtains SoA results, both in terms of accuracy/sparsity and accuracy/FLOPS trade-offs, when progressively increasing the sparsity ratio in a single training cycle. In particular, despite its simplicity, ST-3 favorably compares to the most recent methods, adopting differentiable formulations [42] or bio-inspired neuroregeneration principles [25]. This suggests that the key ingredients for effective sparsification primarily lie in the ability to give the weights the freedom to evolve smoothly across the zero state while progressively increasing the sparsity ratio.

Designing Deep Neural Networks (DNNs) running on edge hardware remains a challenge. Standard designs have been adopted by the community to facilitate the deployment of Neural Network models. However, not much emphasis is put on adapting the network topology to fit hardware constraints. In this paper, we adapt one of the most widely used architectures for mobile hardware platforms, MobileNetV2, and study the impact of changing its topology and applying post-training quantization. We discuss the impact of the adaptations and the deployment of the model on an embedded hardware platform for face detection.

Motivated by the recent trend towards the usage of larger receptive fields for more context-aware neural networks in vision applications, we aim to investigate how large these receptive fields really need to be. To facilitate such study, several challenges need to be addressed, most importantly: (i) We need to provide an effective way for models to learn large filters (potentially as large as the input data) without increasing their memory consumption during training or inference, (ii) the study of filter sizes has to be decoupled from other effects such as the network width or number of learnable parameters, and (iii) the employed convolution operation should be a plug-and-play module that can replace any conventional convolution in a Convolutional Neural Network (CNN) and allow for an efficient implementation in current frameworks. To facilitate such models, we propose to learn not spatial but frequency representations of filter weights as neural implicit functions, such that even infinitely large filters can be parameterized by only a few learnable weights. The resulting neural implicit frequency CNNs are the first models to achieve results on par with the state-of-the-art on large image classification benchmarks while executing convolutions solely in the frequency domain and can be employed within any CNN architecture. They allow us to provide an extensive analysis of the learned receptive fields. Interestingly, our analysis shows that, although the proposed networks could learn very large convolution kernels, the learned filters practically translate into well-localized and relatively small convolution kernels in the spatial domain. NIFF 56x56