Sato

Big Data and Cognitive Computing

Deep neural networks with rate-based neurons have exhibited tremendous progress in the last decade. However, the same level of progress has not been observed in research on spiking neural networks (SNN), despite their capability to handle temporal data, energy-efficiency and low latency. This could be because the benchmarking techniques for SNNs are based on the methods used for evaluating deep neural networks, which do not provide a clear evaluation of the capabilities of SNNs. Particularly, the benchmarking of SNN approaches with regards to energy efficiency and latency requires realization in suitable hardware, which imposes additional temporal and resource constraints upon ongoing projects. This review aims to provide an overview of the current real-world applications of SNNs and identifies steps to accelerate research involving SNNs in the future.

Frontiers in Neuroscience, 2021

Despite the success of Deep Neural Networks-a type of Artificial Neural Network (ANN)-in problem domains such as image recognition and speech processing, the energy and processing demands during both training and deployment are growing at an unsustainable rate in the push for greater accuracy. There is a temptation to look for radical new approaches to these applications, and one such approach is the notion that replacing the abstract neuron used in most deep networks with a more biologicallyplausible spiking neuron might lead to savings in both energy and resource cost. The most common spiking networks use rate-coded neurons for which a simple translation from a pre-trained ANN to an equivalent spike-based network (SNN) is readily achievable. But does the spike-based network offer an improvement of energy efficiency over the original deep network? In this work, we consider the digital implementations of the core steps in an ANN and the equivalent steps in a rate-coded spiking neural network. We establish a simple method of assessing the relative advantages of rate-based spike encoding over a conventional ANN model. Assuming identical underlying silicon technology we show that most rate-coded spiking network implementations will not be more energy or resource efficient than the original ANN, concluding that more imaginative uses of spikes are required to displace conventional ANNs as the dominant computing framework for neural computation.

Neural Networks

Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. However, most supervised SNN training methods, such as conversion from artificial neural networks or direct training with surrogate gradients, require complex computation rather than spike-based operations of spiking neurons during training. In this paper, we study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method, implicit differentiation on the equilibrium state (IDE), for supervised learning with purely spike-based computation, which demonstrates the potential for energy-efficient training of SNNs. Specifically, we introduce ternary spiking neuron couples and prove that implicit differentiation can be solved by spikes based on this design, so the whole training procedure, including both forward and backward passes, is made as event-driven spike computation, and weights are updated locally with two-stage average firing rates. Then we propose to modify the reset membrane potential to reduce the approximation error of

Deep neural networks such as Convolutional Networks (ConvNets) and Deep Belief Networks (DBNs) represent the state-of-the-art for many machine learning and computer vision classification problems. To overcome the large computational cost of deep networks, spiking deep networks have recently been proposed, given the specialized hardware now available for spiking neural networks (SNNs). However, this has come at the cost of performance losses due to the conversion from analog neural networks (ANNs) without a notion of time, to sparsely firing, event-driven SNNs. Here we analyze the effects of converting deep ANNs into SNNs with respect to the choice of parameters for spiking neurons such as firing rates and thresholds. We present a set of optimization techniques to minimize performance loss in the conversion process for ConvNets and fully connected deep networks. These techniques yield networks that outperform all previous SNNs on the MNIST database to date, and many networks here are close to maximum performance after only 20 ms of simulated time. The techniques include using rectified linear units (ReLUs) with zero bias during training, and using a new weight normalization method to help regulate firing rates. Our method for converting an ANN into an SNN enables lowlatency classification with high accuracies already after the first output spike, and compared with previous SNN approaches it yields improved performance without increased training time. The presented analysis and optimization techniques boost the value of spiking deep networks as an attractive framework for neuromorphic computing platforms aimed at fast and efficient pattern recognition.

Scientific Reports, 2021

Neural modelling tools are increasingly employed to describe, explain, and predict the human brain’s behavior. Among them, spiking neural networks (SNNs) make possible the simulation of neural activity at the level of single neurons, but their use is often threatened by the resources needed in terms of processing capabilities and memory. Emerging applications where a low energy burden is required (e.g. implanted neuroprostheses) motivate the exploration of new strategies able to capture the relevant principles of neuronal dynamics in reduced and efficient models. The recent Leaky Integrate-and-Fire with Latency (LIFL) spiking neuron model shows some realistic neuronal features and efficiency at the same time, a combination of characteristics that may result appealing for SNN-based brain modelling. In this paper we introduce FNS, the first LIFL-based SNN framework, which combines spiking/synaptic modelling with the event-driven approach, allowing us to define heterogeneous neuron gro...

IEEE Access

We present an end-to-end trainable modular event-driven neural architecture that uses local synaptic and threshold adaptation rules to perform transformations between arbitrary spatio-temporal spike patterns. The architecture represents a highly abstracted model of existing Spiking Neural Network (SNN) architectures. The proposed Optimized Deep Event-driven Spiking neural network Architecture (ODESA) can simultaneously learn hierarchical spatio-temporal features at multiple arbitrary time scales. ODESA performs online learning without the use of error back-propagation or the calculation of gradients. Through the use of simple local adaptive selection thresholds at each node, the network rapidly learns to appropriately allocate its neuronal resources at each layer for any given problem without using an error measure. These adaptive selection thresholds are the central feature of ODESA, ensuring network stability and remarkable robustness to noise as well as to the selection of initial system parameters. Network activations are inherently sparse due to a hard Winner-Take-All (WTA) constraint at each layer. We evaluate the architecture on existing spatio-temporal datasets, including the spike-encoded IRIS, latency-coded MNIST, Oxford Spike pattern and TIDIGITS datasets, as well as a novel set of tasks based on International Morse Code that we created. These tests demonstrate the hierarchical spatio-temporal learning capabilities of ODESA. Through these tests, we demonstrate ODESA can optimally solve practical and highly challenging hierarchical spatio-temporal learning tasks with the minimum possible number of computing nodes. 17 18 ered in the brain and given our mature understanding of 47 the behaviour of biological neural networks, such evi-48 dence is unlikely to emerge. Second, successful error 49 back-propagation requires global, precise, and repeated prop-50 agation of error measures through all the involved compu-51 tational nodes in a network -beginning from the inputs 52 to the highest layers and back. This has been dubbed the 53 'weight transport problem', as the weights of higher layers 54 have to be made available to the lower layers for successful 55 backpropagation of error values [4]. Again, no evidence for 56 such processes has been, or is likely to be, found. The third 57 and most crucial pillar of error back-propagation is the dif-58 ferentiability requirement for all the constituent components 59 of a given network. Indeed, it is this very aspect of the 60 error back-propagation algorithm which makes it difficult to 61 use on non-differentiable data domains like spatio-temporal 62 spikes patterns which the brains use as the primary mode of 63 computation and communication. This leads to the biggest 64 problem in the use of error back-propagation in computa-65 tional neuroscience namely, the credit-assignment problem. 66 There have been multiple attempts at approximating error 67 back-propagation and applying gradient descent to SNN 68 architectures. SpikeProp [5] was among the first works to 69 derive a supervised learning rule for SNNs from the error 70 back-propagation algorithm. Tempotron [6] was introduced 71 in which the error back-propagation was applied by defining 72 loss functions based on the maximum voltage and the thresh-73 old voltage of the output neurons. Chronotron [7] was intro-74 duced as an improvement over Tempotron by using a new 75 distance metric between the predicted and target spike trains. 76 More recent works applied error back-propagation to SNN 77 architectures by using different surrogate gradients for the 78 hard thresholding activation functions of the spiking neurons 79 [8], [9], [10], [11]. However, they don't address how biology 80 can realize the computation of gradients and their access 81 to the neurons involved in the computation. Furthermore, 82 we don't have evidence on how batching of data happens 83 in biology, and most of the gradient descent approaches rely 84 on batching the data. Despite the lack of bio-plausibility, the 85 error back-propagation based approaches have been adopted 86 in computational neuroscience as useful alternative tools to 87 discover the required connectivity in SNNs for a given spe-88 cific task [12], [13], [14]. 89 Feedback alignment has been used as an alternative to error 90 back-propagation for SNNs in [15] to solve the 'weight trans-91 port' problem. Feedback alignment shows that multiplying 92 errors by random synaptic weights is enough for effective 93 error back-propagation without requiring a precise symmet-94 ric backward connectivity pattern. There have been parallel 95 investigations of more bio-plausible local learning rules for 96 SNNs which do not require access to the weights of other 97 neurons in the network. This set of learning rules for SNNs 98 can be characterized as synaptic plasticity rules which use 99 Spike-Timing-Dependent Plasticity (STDP) in some form. 100 STDP rules have more commonly been used for extracting 101 paper, we propose to use these adaptive selection thresholds 158 for multi-layer supervised learning and propose our method 159 as a simple solution to the credit assignment problem. Our 160 proposed method is the first to not require the transport of 161 weights across neurons in the network or random connections 162 for error propagation. We achieve all feedback to earlier 163 layers using precisely timed binary attention signals which 164 signal ''reward'' and ''punishment'' of the recently active 165 neurons. 166 II. BACKGROUND AND RELATED WORK 167 A. TIME SURFACES 168 Tapson et al. [47] proposed the use of exponentially decaying 169 kernels for processing event data produced by neuromorphic 170 vision sensors. We shall use the terminology introduced by 171 Lagorce et al. [45] and refer to these kernels as time surfaces. 172 The time surface is the trace of events per each input channel 173 which is updated only when an event arrives at a channel. An event from an event-based sensor can be described as: Equation ( ) describes an event from a pixel location 177 (x i , y i ) as the coordinates on the sensor, with polarity p i , 178 arriving at time t i . A similar notation can be used to represent 179 a spike as an event: 180 181 Equation (2) represents a spike s i from channel c i at time t i . We can use the time surfaces to keep the trace of spikes from 183 a spiking source. The exponential time surface S t [i] of a channel i at time t 185 with a time constant of τ can be calculated as follows: 3) attempting to demonstrate biological plausibility through 259 detailed phenomenological modelling from the voltage-gated 260 ion channels to the delays at the neuronal synapses tend to 261 be limited in their performance and utility in the context of 262 challenging machine learning tasks. The computational cost 263 of these models increases with the bio-plausibility of the 264 model. Different neuronal models have been proposed which 265 approximate and abstract the details of these complexities 266 with easy-to-handle mathematical and probabilistic models 267 [50], [51], [52], [53]. Leaky-Integrate and Fire (LIF) neu-268 ron model [49] and specifically the Spike Response Model 269 (SRM) [54] are among the most popular choices of neuron 270 models in the SNNs, even though the degree to which they 271 explain the neuronal dynamics is limited compared to other 272 models like Hodgkin-Huxley [55] or Izhkevich [52] mod-273 els. Their vast adoption can be attributed to their analytical 274 tractability and computational simplicity compared to other 275 neuronal models. But even the SNN models which use sim-276 pler neuronal models like LIF or Adaptive Leaky-Integrate 277 and Fire (ALIF) neurons [56] require additional complexities 278 such as Excitatory-Inhibitory Balance, and the right amount 279 of lateral excitation and inhibition to instil behaviours like 280 WTA. These complex processes make it difficult to scale 281 up the simulations of the multi-layered SNNs and limit the 282 exploration of broader system-level learning mechanisms of 283 the SNNs as there are a lot of variables in the system. In the 284 same way, that time surfaces represent simplified hardware 285 friendly abstractions of the EPSP, the FEAST network can be 286 best understood as a highly abstracted, functionally equiva-287 lent, modular implementation of a well-balanced excitatory 288 SNN with inhibitory feedback leading to a winner take all 289 operation at a single layer. In this way, a FEAST layer rep-290 resents a neuron group. Picking only one winner in each 291 layer of FEAST for any input event is a proxy for hard WTA 292 motif in a neuron group, without requiring any forms of 293 inhibition. Simpler and computationally easier abstract SNN 294 models like FEAST can help us explore more system-level 295 learning rules in SNNs without having to worry about prob-296 lems like achieving EI balance and promoting or removing 297 oscillations in the networks. Just like Address-Event Rep-298 resentations (AER) being used in Neuromorphic hardware 299 to facilitate the communication in SNNs, we can use novel 300 abstractions like FEAST to explore the space of local learning 301 rules in Spiking Neural Architectures. Continuing in this 302 approach and extending it, the Optimized Deep Event-driven 303 Spiking neural network Architecture (ODESA) introduced in 304 this paper, represents a method to locally train hierarchies of ''1'', the network also has to learn the position of the same 788 symbol in the sequence. We used a two-layered ODESA 789 network, one layer to learn the symbols (0 and 1), and the 790 second layer, which is the output layer, to learn the sequence 791 of the symbols. The output layer has two neurons for the 792 two classes. The ODESA network can learn an intermedi-793 ate representation of the symbol ''1'' without relying on an 794 explicit supervisory signal for symbol ''1'' due to the local 795...

Frontiers in Neuroscience, 2021

The development of brain-inspired neuromorphic computing architectures as a paradigm for Artificial Intelligence (AI) at the edge is a candidate solution that can meet strict energy and cost reduction constraints in the Internet of Things (IoT) application areas. Toward this goal, we present μBrain: the first digital yet fully event-driven without clock architecture, with co-located memory and processing capability that exploits event-based processing to reduce an always-on system's overall energy consumption (μW dynamic operation). The chip area in a 40 nm Complementary Metal Oxide Semiconductor (CMOS) digital technology is 2.82 mm2 including pads (without pads 1.42 mm2). This small area footprint enables μBrain integration in re-trainable sensor ICs to perform various signal processing tasks, such as data preprocessing, dimensionality reduction, feature selection, and application-specific inference. We present an instantiation of the μBrain architecture in a 40 nm CMOS digital...

2017 International Joint Conference on Neural Networks (IJCNN), 2017

In recent years, multiple neuromorphic architectures have been designed to execute cognitive applications that deal with image and speech analysis. These architectures have followed one of two approaches. One class of architectures is based on machine learning with artificial neural networks. A second class is focused on emulating biology with spiking neuron models, in an attempt to eventually approach the brain's accuracy and energy efficiency. A prominent example of the second class is IBM's TrueNorth processor that can execute large spiking networks on a low-power tiled architecture, and achieve high accuracy on a variety of tasks. However, as we show in this work, there are many inefficiencies in the TrueNorth design. We propose a new architecture, INXS, for spiking neural networks that improves upon the computational efficiency and energy efficiency of the TrueNorth design by 3,129× and 10× respectively. The architecture uses memristor crossbars to compute the effects of input spikes on several neurons in parallel. Digital units are then used to update neuron state. We show that the parallelism offered by crossbars is critical in achieving high throughput and energy efficiency. Network Type Accuracy Throughput (TOPs/s) Energy per operation (pJ/op) MNIST CIFAR-10

Frontiers in Neuroscience, 2020

Spiking Neural Networks (SNNs) may offer an energy-efficient alternative for implementing deep learning applications. In recent years, there have been several proposals focused on supervised (conversion, spike-based gradient descent) and unsupervised (spike timing dependent plasticity) training methods to improve the accuracy of SNNs on large-scale tasks. However, each of these methods suffer from scalability, latency and accuracy limitations. In this paper, we propose novel algorithmic techniques of modifying the SNN configuration with backward residual connections, stochastic softmax and hybrid artificial-and-spiking neuronal activations to improve the learning ability of the training methodologies to yield competitive accuracy, while, yielding large efficiency gains over their artificial counterparts. Note, artificial counterparts refer to conventional deep learning/artificial neural networks. Our techniques apply to VGG/Residual architectures, and are compatible with all forms of training methodologies. Our analysis reveals that the proposed solutions yield near state-of-the-art accuracy with significant energy-efficiency and reduced parameter overhead translating to hardware improvements on complex visual recognition tasks, such as, CIFAR10, Imagenet datatsets.

IEEE Transactions on Parallel and Distributed Systems

Compiler frameworks are crucial for the widespread use of FPGA-based deep learning accelerators. They allow researchers and developers, who are not familiar with hardware engineering, to harness the performance attained by domain-specific logic. There exists a variety of frameworks for conventional artificial neural networks. However, not much research effort has been put into the creation of frameworks optimized for spiking neural networks (SNNs). This new generation of neural networks becomes increasingly interesting for the deployment of AI on edge devices, which have tight power and resource constraints. Our end-to-end framework E3NE automates the generation of efficient SNN inference logic for FPGAs. Based on a PyTorch model and user parameters, it applies various optimizations and assesses trade-offs inherent to spike-based accelerators. Multiple levels of parallelism and the use of an emerging neural encoding scheme result in an efficiency superior to previous SNN hardware implementations. For a similar model, E3NE uses less than 50% of hardware resources and 20% less power, while reducing the latency by an order of magnitude. Furthermore, scalability and generality allowed the deployment of the large-scale SNN models AlexNet and VGG.