The Parallel HTM Spatial Pooler

Computers & Electrical Engineering, 1993

A~tract--In executing tasks involving intelligent information processing, the human brain performs better than the digital computer. The human brain derives its power from a large number [O(1011)] of neurons which are interconnected by a dense interconnection network [O(105) connections per neuron]. Artificial neural network (ANN) paradigms adopt the structure of the brain to try to emulate the intelligent information processing methods of the brain. ANN techniques are being employed to solve problems in areas such as pattern recognition, and robotic processing. Simulation of ANNs involves implementation of large number of neurons and a massive interconnection network. In this paper, we discuss various simulation models of ANNs and their implementation on distributed memory systems. Our investigations reveal that communication-efficient networks of distributed memory systems perform better than other topologies in implementing ANNs.

2011

While a number of neuromorphic studies have been based on understanding and building the brain in software and hardware, a recent theory has been presented from a high level, top down approach, with the view of understanding how the human brain performs higher reasoning, and then designing software infrastructure based on that theorynamely Hierarchical Temporal Memory (HTM). Current opinion raises question marks as to whether this theory is sound, feasible, coherent, logical and correct. This thesis aims to, if not answer some of those questions, at least get closer to understanding the brain and therefore implementing HTM theory in software for further use in an embedded environment. This paper begins by reviewing the discovery of the sensory input and chemical makeup of the signals in the brain, and then introduces mathematical abstraction to form a software program node. The final output of this thesis is a Hierarchical Temporal node, designed in software, and demonstrating learning using pseudo-random input sensory data and the spatial temporal framework; it verifies that the program operates as it should, and suggests how to create a VLSI circuit implementation of the software and ultimately a silicon chip that can be used in intelligent machines and prostheses. As this dissertation has taken shape, so has the research backing it. Two fundamental papers, used as references, were published during the writing; lack of published research on HTM has both been a hindrance and a driving factor.

2009

Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 2012

Progress on large-scale simulation of neural models depends in part on the availability of suitable hardware and software architectures. Heterogeneous hardware computing platforms are becoming increasingly popular as substrates for general-purpose simulation. On the other hand, recent work highlights that certain constraints on neural models must be imposed on neural and synaptic dynamics in order to take advantage of such systems. In this paper we focus on constraints related to learning in a simple visual system and those imposed by a new neural simulator for heterogeneous hardware systems, CogExMachina (Cog).

ArXiv, 2015

In the decade since Jeff Hawkins proposed Hierarchical Temporal Memory (HTM) as a model of neocortical computation, the theory and the algorithms have evolved dramatically. This paper presents a detailed description of HTM's Cortical Learning Algorithm (CLA), including for the first time a rigorous mathematical formulation of all aspects of the computations. Prediction Assisted CLA (paCLA), a refinement of the CLA is presented, which is both closer to the neuroscience and adds significantly to the computational power. Finally, we summarise the key functions of neocortex which are expressed in paCLA implementations.

Intelligent Computing Systems, 2018

At this time, great effort is being directed toward developing problem-solving technology that mimic human cognitive processes. Research has been done to develop object recognition using Computer Vision for daily tasks such as secure access, traffic management, and robotic behavior. For this research, four different machine learning algorithms have been developed to overcome the computer vision problem of object recognition. Hierarchical temporal memory (HTM) is an emerging technology based on biological methods of the human cortex to learn patterns. This research applied an HTM algorithm to images (video sequences) in order to compare this technique against two others: support vector machines (SVM) and artificial neural networks (ANN). It was concluded that HTM was the most effective.

Journal of Aerospace Computing Information and Communication, 2008

There is renewed interest in computational intelligence, due to advances in algorithms, neuroscience, and computer hardware. In addition there is enormous interest in autonomous vehicles (air, ground, and sea) and robotics, which need significant onboard intelligence. Work in this area could not only lead to better understanding of the human brain but also very useful engineering applications. The functioning of the human brain is not well understood, but enormous progress has been made in understanding it and, in particular, the neocortex. There are many reasons to develop models of the brain. Artificial Neural Networks (ANN), one type of model, can be very effective for pattern recognition, function approximation, scientific classification, control, and the analysis of time series data. ANNs often use the back-propagation algorithm for training, and can require large training times especially for large networks, but there are many other types of ANNs. Once the network is trained for a particular problem, however, it can produce results in a very short time. Parallelization of ANNs could drastically reduce the training time. An object-oriented, massively-parallel ANN (Artificial Neural Network) software package SPANN (Scalable Parallel Artificial Neural Network) has been developed and is described here. MPI was used to parallelize the C++ code. Only the neurons on the edges of the domains were involved in communication, in order to reduce the communication costs and maintain scalability. The back-propagation algorithm was used to train the network. In preliminary tests, the software was used to identify character sets. The code correctly identified all the characters when adequate training was used in the network. The code was run on up to 500 Intel Itanium processors with 25,000 neurons and more than 2 billion neuron weights. Various comparisons in training time, forward propagation time, and error reduction were also made.

Recently the improved bag of features (BoF) model with locality-constrained linear coding (LLC) and spatial pyramid matching (SPM) achieved state-of-the-art performance in image classification. However, only adopting SPM to exploit spatial information is not enough for satisfactory performance. In this paper, we use hierarchical temporal memory (HTM) cortical learning algorithms to extend this LLC & SPM based model. HTM regions consist of HTM cells are constructed to spatial pool the LLC codes. Each cell receives a subset of LLC codes, and adjacent subsets are overlapped so that more spatial information can be captured. Additionally, HTM cortical learning algorithms have two processes: learning phase which make the HTM cell only receive most frequent LLC codes, and inhibition phase which ensure that the output of HTM regions is sparse. The experimental results on Caltech 101 and UIUC-Sport dataset show the improvement on the original LLC & SPM based model.

The 2013 International Joint Conference on Neural Networks (IJCNN), 2013

Marching along the DARPA SyNAPSE roadmap, IBM unveils a trilogy of innovations towards the TrueNorth cognitive computing system inspired by the brain's function and efficiency. The non-von Neumann nature of the TrueNorth architecture necessitates a novel approach to efficient system design. To this end, we have developed a set of abstractions, algorithms, and applications that are natively efficient for TrueNorth. First, we developed repeatedly-used abstractions that span neural codes (such as binary, rate, population, and time-to-spike), long-range connectivity, and short-range connectivity. Second, we implemented ten algorithms that include convolution networks, spectral content estimators, liquid state machines, restricted Boltzmann machines, hidden Markov models, looming detection, temporal pattern matching, and various classifiers. Third, we demonstrate seven applications that include speaker recognition, music composer recognition, digit recognition, sequence prediction, collision avoidance, optical flow, and eye detection. Our results showcase the parallelism, versatility, rich connectivity, spatio-temporality, and multi-modality of the TrueNorth architecture as well as compositionality of the corelet programming paradigm and the flexibility of the underlying neuron model.

ArXiv, 2016

Hierarchical Temporal Memory (HTM) is a biomimetic machine learning algorithm imbibing the structural and algorithmic properties of the neocortex. Two main functional components of HTM that enable spatio-temporal processing are the spatial pooler and temporal memory. In this research, we explore a scalable hardware realization of the spatial pooler closely coupled with the mathematical formulation of spatial pooler. This class of neuromorphic algorithms are advantageous in solving a subset of the future engineering problems by extracting nonintuitive patterns in complex data. The proposed architecture, Non-volatile HTM (NVHTM), leverages large-scale solid state flash memory to realize a optimal memory organization, area and power envelope. A behavioral model of NVHTM is evaluated against the MNIST dataset, yielding 91.98% classification accuracy. A full custom layout is developed to validate the design in a TSMC 180nm process. The area and power profile of the spatial pooler are 30....