Backpropagation through time: what it does and how to do it

International Journal of Circuit Theory and Applications, 2006

In this paper the problem of learning spatiotemporal behaviour with cellular neural networks is analysed and a novel method is proposed to approach the problem. The basis for this method is found in trajectory learning with recurrent neural networks. Despite of similarities, the two learning problems have underling differences which make non-trivial a direct mapping into the problem at hand. In order to solve the problem, a new cost function is proposed, which also assimilates time instants as parameters to be optimized. As a consequence, it does not force the desired spatiotemporal behaviour to be learned in its original speed, and thus different speed versions of the desired behaviour are allowed to be learned; hence, also providing a promising direction for increasing the speed of existing applications. Learning examples are presented for different classes of spatiotemporal dynamics including spiral autowaves. Results of simulation and on-chip learning show that the proposed approach is able to learn these dynamics with cellular neural networks. Copyright © 2006 John Wiley & Sons, Ltd.

Lecture Notes in Computer Science, 2010

Different varieties of artificial neural networks have proved their power in several pattern recognition problems, particularly feedforward neural networks. Nevertheless, these kinds of neural networks require of several neurons and layers in order to success when they are applied to solve non-linear problems. In this paper is shown how a spiking neuron can be applied to solve different linear and non-linear pattern recognition problems. A spiking neuron is stimulated during T ms with an input signal and fires when its membrane potential reaches a specific value generating an action potential (spike) or a train of spikes. Given a set of input patterns belonging to K classes, each input pattern is transformed into an input signal, then the spiking neuron is stimulated during T ms and finally the firing rate is computed. After adjusting the synaptic weights of the neuron model, we expect that input patterns belonging to the same class generate almost the same firing rate and input patterns belonging to different classes generate firing rates different enough to discriminate among the different classes. At last, a comparison between a feed-forward neural network and a spiking neuron is presented when they are applied to solve non-linear and real object recognition problems.

International Journal of Intelligent Systems, 2004

,QGH[ 7HUPVGHOD\HG IHHGEDFN FRQQHFWLRQV IHHGIRUZDUG QHXUDO QHWZRUNV PHPRU\ HIIHFWV , ,IUSP9V8UDPI 7C@ DUWLILFLDO QHXUDO QHWZRUNV KDYH WKH JUHDWHVW SRWHQWLDO LQ WKH FRQWURO ILHOG GXH WR WKHLU FDSDELOLW\ RI DSSUR[LPDWLQJ DQ\ QRQOLQHDU PDSSLQJV PDVVLYH SDUDOOHO SURFHVVLQJ RI LQIRUPDWLRQ DQG OHDUQLQJ DQG DGDSWDWLRQ 7KH DELOLW\ RI WKH DUWLILFLDO QHXUDO QHWZRUNV RI PDSSLQJ QRQOLQHDU UHODWLRQVKLSV ZLWKRXW D SULRUL LQIRUPDWLRQ DERXW WKH SURFHVV PRGHO PDNHV WKHP H[WUHPHO\ XVHIXO LQ V\VWHP LGHQWLILFDWLRQ >@ >@ DQG >@ $PRQJ WKH YDULHW\ RI QHXUDO QHWZRUN DUFKLWHFWXUHV WKH IHHGIRUZDUG QHXUDO QHWZRUNV DUH E\ IDU PRVW IUHTXHQWO\ DSSOLHG LQ FRQWURO SUREOHPV 7KH\ FRQVLVW RI D OD\HU RI LQSXW QHXURQV D OD\HU RI RXWSXW QHXURQV DQG RQH RU VHYHUDO KLGGHQ LQWHUPHGLDWH OD\HUV ,Q VXFK W\SH RI QHXUDO QHWZRUN WKH LQIRUPDWLRQ SURFHVVHV IURP LQSXW WR RXWSXW LQ IRUZDUG GLUHFWLRQ 7KH QHXUDO QHWZRUNV DUH WUDLQHG E\

Sensors, 2014

Grinding is an advanced machining process for the manufacturing of valuable complex and accurate parts for high added value sectors such as aerospace, wind generation, etc. Due to the extremely severe conditions inside grinding machines, critical process variables such as part surface finish or grinding wheel wear cannot be easily and cheaply measured on-line. In this paper a virtual sensor for on-line monitoring of those variables is presented. The sensor is based on the modelling ability of Artificial Neural Networks (ANNs) for stochastic and non-linear processes such as grinding; the selected architecture is the Layer-Recurrent neural network. The sensor makes use of the relation between the variables to be measured and power consumption in the wheel spindle, which can be easily measured. A sensor calibration methodology is presented, and the levels of error that can be expected are discussed. Validation of the new sensor is carried out by comparing the sensor's results with actual measurements carried out in an industrial grinding machine. Results show excellent estimation performance for both wheel wear and surface roughness. In the case of wheel wear, the absolute error is within the range of microns (average value 32 μm). In the case of surface finish, the absolute error is well below R a 1 μm (average value 0.32μm). The present approach can be easily generalized to other grinding operations.

Quaternary Research, 2007

A unique 800-yr-long record of annual temperatures and precipitation over the south of western Siberia has been reconstructed from the bottom sediments of Teletskoye Lake, Altai Mountains using an X-ray fluorescence scanner (XRF) providing 0.1-mm resolution timeseries of elemental composition and X-ray density (XRD). Br content appears to be broadly correlative with mean annual temperature variations because of changes in catchment vegetation productivity. Sr/Rb ratio reflects the proportion of the unweathered terrestrial fraction. XRD appears to reflect water yield regime and sediment flux. Sedimentation is rather continuous because annual clastic supply and deposited mass are the same. The artificial neural networks method was applied to convert annual sedimentary time-series of XRD, Br content, and Sr/Rb ratio to annual records of temperature and precipitation using a transfer function. Comparison of these reconstructed Siberian records with the annual record of air temperature for the Northern Hemisphere shows similar trends in climatic variability over the past 800 yr. Estimated harmonic oscillations of temperature and precipitation values for both historical and reconstructed periods reveal subdecadal cyclicity.

Systems Science & Control Engineering, 2015

Nonlinearities in system dynamics and the multivariable nature of processes offer a stiff challenge in designing predictive controllers that improve process performance in industries. This investigation presents a recurrent neuro fuzzy network (RNFN) model for a nonlinear multivariable system in process industries and a methodology to design model-predictive controllers (MPCs) using the proposed model. The RNFN model combines the learning features of artificial neural networks with human cognition capabilities of fuzzy systems. Therefore, RNFN leads to a modelling framework that has the ability not only to learn the model parameters, but also makes decision on operating region of the nonlinear model depending on the input-output data. Furthermore, the recurrent structure and the introduction of a memory unit between the fuzzy inference and fuzzification layer enhance the prediction capability due to the use of past input-output data, making the model more suitable for designing predictive controllers. Next, the MPC design methodology that exploits the advantages of the RNFN model to optimize the control moves is presented. The proposed MPC uses the gradient descent algorithm to minimize the control moves as against the traditional state-space approaches that require complex computations and solvers. Therefore, implementing the proposed MPC in embedded hardware becomes easier. The proposed modelling framework and the MPC design methodology are illustrated using experiments on a laboratory-scale quadruple tank. Our experiments show that the proposed RNFN-based MPC performs better than the neuro fuzzy network-based MPC for both servo and regulatory responses.

Hydrological Sciences Journal, 2014

dates are example text only)

Water Resources Management, 2015

Transportation Research Record: Journal of the Transportation Research Board, 2013

The paper compares univariate and multivariate neural network and autoregressive time series models with application to short-term forecasting of freeway speeds. The developed models are evaluated with respect to temporal data resolution, prediction accuracy, and quality of fit using statistical tests. Results indicate that neural networks provide -by and large -more accurate predictions than classical statistical approaches particularly for finer data resolutions. Evaluation of model fit indicates that, in contrast to vector autoregressive models, Neural Networks may also provide unbiased predictions. Overall, our findings clearly suggest the need to jointly consider statistical and Neural Network models in order to develop more efficient prediction models.

Lecture Notes in Computer Science, 2009

Mathematical Problems in Engineering, 2015

Recurrent neural network (RNN) has been widely used as a tool in the data classification. This network can be educated with gradient descent back propagation. However, traditional training algorithms have some drawbacks such as slow speed of convergence being not definite to find the global minimum of the error function since gradient descent may get stuck in local minima. As a solution, nature inspired metaheuristic algorithms provide derivative-free solution to optimize complex problems. This paper proposes a new metaheuristic search algorithm called Cuckoo Search (CS) based on Cuckoo bird's behavior to train Elman recurrent network (ERN) and back propagation Elman recurrent network (BPERN) in achieving fast convergence rate and to avoid local minima problem. The proposed CSERN and CSBPERN algorithms are compared with artificial bee colony using BP algorithm and other hybrid variants algorithms. Specifically, some selected benchmark classification problems are used. The simulation results show that the computational efficiency of ERN and BPERN training process is highly enhanced when coupled with the proposed hybrid method.

Neural Computation, 2000

A large class of nonlinear dynamic adaptive systems such as dynamic recurrent neural networks can be effectively represented by signal flow graphs (SFGs). By this method, complex systems are described as a general connection of many simple components, each of them implementing a simple one-input, one-output transformation, as in an electrical circuit. Even if graph representations are popular in the neural network community, they are often used for qualitative description rather than for rigorous representation and computational purposes. In this article, a method for both on-line and batch-backward gradient computation of a system output or cost function with respect to system parameters is derived by the SFG representation theory and its known properties. The system can be any causal, in general nonlinear and time-variant, dynamic system represented by an SFG, in particular any feedforward, time-delay, or recurrent neural network. In this work, we use discrete-time notation, but the same theory holds for the continuous-time case. The gradient is obtained in a straightforward way by the analysis of two SFGs, the original one and its adjoint (obtained from the first by simple transformations), without the complex chain rule expansions of derivatives usually employed. This method can be used for sensitivity analysis and for learning both off-line and on-line. On-line learning is particularly important since it is required by many real applications, such as digital signal processing, system identification and control, channel equalization, and predistortion.

Computational Intelligence, 2014

Many data mining and data analysis techniques operate on dense matrices or complete tables of data. Realworld data sets, however, often contain unknown values. Even many classification algorithms that are designed to operate with missing values still exhibit deteriorated accuracy. One approach to handling missing values is to fill in (impute) the missing values. In this paper, we present a technique for unsupervised learning called Unsupervised Backpropagation (UBP), which trains a multi-layer perceptron to fit to the manifold sampled by a set of observed point-vectors. We evaluate UBP with the task of imputing missing values in datasets, and show that UBP is able to predict missing values with significantly lower sum-squared error than other collaborative filtering and imputation techniques. We also demonstrate with 24 datasets and 9 supervised learning algorithms that classification accuracy is usually higher when randomly-withheld values are imputed using UBP, rather than with other methods.

Energies

Conventional sparse spike deconvolution algorithms that are based on the iterative shrinkage-thresholding algorithm (ISTA) are widely used. The aim of this type of algorithm is to obtain accurate seismic wavelets. When this is not fulfilled, the processing stops being optimum. Using a recurrent neural network (RNN) as deep learning method and applying backpropagation to ISTA, we have developed an RNN-like ISTA as an alternative sparse spike deconvolution algorithm. The algorithm is tested with both synthetic and real seismic data. The algorithm first builds a training dataset from existing well-logs seismic data and then extracts wavelets from those seismic data for further processing. Based on the extracted wavelets, the new method uses ISTA to calculate the reflection coefficients. Next, inspired by the backpropagation through time (BPTT) algorithm, backward error correction is performed on the wavelets while using the errors between the calculated reflection coefficients and the ...

Water Resources Management, 2014

Your article is protected by copyright and all rights are held exclusively by Springer Science +Business Media Dordrecht. This e-offprint is for personal use only and shall not be selfarchived in electronic repositories. If you wish to self-archive your article, please use the accepted manuscript version for posting on your own website. You may further deposit the accepted manuscript version in any repository, provided it is only made publicly available 12 months after official publication or later and provided acknowledgement is given to the original source of publication and a link is inserted to the published article on Springer's website. The link must be accompanied by the following text: "The final publication is available at link.springer.com".

Scientific Reports

In recent years artificial neural networks achieved performance close to or better than humans in several domains: tasks that were previously human prerogatives, such as language processing, have witnessed remarkable improvements in state of the art models. One advantage of this technological boost is to facilitate comparison between different neural networks and human performance, in order to deepen our understanding of human cognition. Here, we investigate which neural network architecture (feedforward vs. recurrent) matches human behavior in artificial grammar learning, a crucial aspect of language acquisition. Prior experimental studies proved that artificial grammars can be learnt by human subjects after little exposure and often without explicit knowledge of the underlying rules. We tested four grammars with different complexity levels both in humans and in feedforward and recurrent networks. Our results show that both architectures can “learn” (via error back-propagation) the...

Journal of Marine Science and Engineering

Constructing offshore and coastal structures with the highest level of stability and lowest cost, as well as the prevention of faulty risk, is the desired plan that stakeholders seek to obtain. The successful construction plans of such projects mostly rely on well-analyzed and modeled metocean data that yield high prediction accuracy for the ocean environmental conditions including waves and wind. Over the past decades, planning and designing coastal projects have been accomplished by traditional static analytic, which requires tremendous efforts and high-cost resources to validate the data and determine the transformation of metocean data conditions. Therefore, the wind plays an essential role in the oceanic atmosphere and contributes to the formation of waves. This paper proposes an enhanced weight-optimized neural network based on Sine Cosine Algorithm (SCA) to accurately predict the wave height. Three neural network models named: Long Short-Term Memory (LSTM), Vanilla Recurrent ...

Journal of Aircraft, 2006

An artificial neural-network model for predicting the failure rate of De Havilland Dash-8 airplane tires utilizing the two-layered feedforward back-propagation algorithm as a learning rule is developed. The inputs to the neural network are independent variables, and the output is the failure rate of the tires. Six years of data are used for model building and validation. Model validation, which reflects the suitability of the model for future prediction, is performed by comparing the predictions of the model with that of the Weibull regression model. The results show that the failure rate predicted by the artificial neural network more closely agrees with the actual data than the failure rate predicted by the Weibull model.

Chaos: An Interdisciplinary Journal of Nonlinear Science

Neural networks, and in general machine learning techniques, have been widely employed in forecasting time series and more recently in predicting spatial-temporal signals. All of these approaches involve some kind of feature selection regarding what past data and what neighbor data to use for forecasting. In this article, we show extensive empirical evidence on how to independently construct the optimal feature selection or input representation used by the input layer of a feed forward neural network for the purpose of forecasting spatial-temporal signals. The approach is based on results from the dynamical systems theory, namely, nonlinear embedding theorems. We demonstrate it for a variety of spatial-temporal signals and show that the optimal input layer representation consists of a grid, with spatial-temporal lags determined by the minimum of the mutual information of the spatial-temporal signals and the number of points taken in space-time decided by the embedding dimension of the signal. We present evidence of this proposal by running a Monte Carlo simulation of several combinations of input layer feature designs and show that the one predicted by the nonlinear embedding theorems seems to be optimal or close to being optimal. In total, we show evidence in four unrelated systems: a series of coupled Hénon maps, a series of coupled ordinary di erential equations (Lorenz-96) phenomenologically modeling atmospheric dynamics, the Kuramoto-Sivashinsky equation, a partial di erential equation used in studies of instabilities in laminar ame fronts, and nally real physical data from sunspot areas in the Sun (in latitude and time) from 1874 to 2015. These four examples cover the range from simple toy models to complex nonlinear dynamical simulations and real data. Finally, we also compare our proposal against alternative feature selection methods and show that it also works for other machine learning forecasting models.

Advances in Physics: X

Technological advancements enable collecting vast data, i.e., Big Data, in science and industry including biomedical field. Increased computational power allows expedient analysis of collected data using statistical and machine-learning approaches. Historical data incompleteness problem and curse of dimensionality diminish practical value of pure data-driven approaches, especially in biomedicine. Advancements in deep learning (DL) frameworks based on deep neural networks (DNN) improved accuracy in image recognition, natural language processing, and other applications yet severe data limitations and/or absence of transfer-learning-relevant problems drastically reduce advantages of DNNbased DL. Our earlier works demonstrate that hierarchical data representation can be alternatively implemented without NN, using boosting-like algorithms for utilization of existing domain knowledge, tolerating significant data incompleteness, and boosting accuracy of low-complexity models within the classifier ensemble, as illustrated in physiological-data analysis. Beyond obvious use in initialfactor selection, existing simplified models are effectively employed for generation of realistic synthetic data for later DNN pre-training. We review existing machine learning approaches, focusing on limitations caused by training-data incompleteness. We outline our hybrid framework that leverages existing domain-expert models/ knowledge, boosting-like model combination, DNNbased DL and other machine learning algorithms for drastic reduction of training-data requirements. Applying this framework is illustrated in context of analyzing physiological data.

Journal of Marine Science and Engineering

Harmful algal blooms (HABs), events that kill fish, impact human health in multiple ways, and contaminate water supplies, have increased in frequency, magnitude, and impacts in numerous marine and freshwaters around the world. Blooms of the toxic dinoflagellate Karenia brevis have resulted in thousands of tons of dead fish, deaths to many other marine organisms, numerous respiratory-related hospitalizations, and tens to hundreds of millions of dollars in economic damage along the West Florida coast in recent years. Four types of machine learning algorithms, Support Vector Machine (SVM), Relevance Vector Machine (RVM), Naïve Bayes classifier (NB), and Artificial Neural Network (ANN), were developed and compared in their ability to predict these blooms. Comparing the 21 year monitoring dataset of K. brevis abundance, RVM and NB were found to have better skills in bloom prediction than the other two approaches. The importance of upwelling-favorable northerly winds in increasing K. brev...

Proceedings of the 2020 Genetic and Evolutionary Computation Conference

Transfer learning entails taking an articial neural network (ANN) that is trained on a source dataset and adapting it to a new target dataset. While this has been shown to be quite powerful, its use has generally been restricted by architectural constraints. Previously, in order to reuse and adapt an ANN's internal weights and structure, the underlying topology of the ANN being transferred across tasks must remain mostly the same while a new output layer is attached, discarding the old output layer's weights. This work introduces network-aware adaptive structure transfer learning (N-ASTL), an advancement over prior eorts to remove this restriction. N-ASTL utilizes statistical information related to the source network's topology and weight distribution in order to inform how new input and output neurons are to be integrated into the existing structure. Results show improvements over prior state-of-the-art, including the ability to transfer in challenging real-world datasets not previously possible and improved generalization over RNNs without transfer. CCS CONCEPTS • Computer systems organization → Neural networks; • Computing methodologies → Transfer learning; • Theory of computation → Evolutionary algorithms;

Adaptive Behavior

Learning to make decisions in partially observable environments is a notorious problem that requires a complex representation of controllers. In most work, the controllers are designed as a non-linear mapping from a sequence of temporal observations to actions. These problems can, in principle, be formulated as a partially observable Markov decision process whose policy can be parameterised through the use of recurrent neural networks. In this paper, we will propose an alternative framework that (a) uses the Long-Short-Term-Memory (LSTM) Encoder-Decoder framework to learn an internal state representation for historical observations and then (b) integrates it into existing recurrent policy models to improve the task performance. The LSTM Encoder encodes a history of observations as input into a representation of internal states. The LSTM Decoder can perform two alternative decoding tasks: predicting the same input observation sequence or predicting future observation sequences. The f...

Machine Learning and Knowledge Discovery in Databases

Searching images with multi-attribute queries shows practical significance in various real world applications. The key problem in this task is how to effectively and efficiently learn from the conjunction of query attributes. In this paper, we propose Attribute Conjunction Recurrent Neural Network (AC-RNN) to tackle this problem. Attributes involved in a query are mapped into the hidden units and combined in a recurrent way to generate the representation of the attribute conjunction, which is then used to compute a multi-attribute classifier as the output. To mitigate the data imbalance problem of multi-attribute queries, we propose a data weighting procedure in attribute conjunction learning with small positive samples. We also discuss on the influence of attribute order in a query and present two methods based on attention mechanism and ensemble learning respectively to further boost the performance. Experimental results on aPASCAL, ImageNet Attributes and LFWA datasets show that our method consistently and significantly outperforms the other comparison methods on all types of queries. The software related to this paper is available at https://github.com/GriffinLiang/ AC-RNN.

Sensors

Sleep apnea is a common sleep disorder that causes repeated breathing interruption during sleep. The performance of automated apnea detection methods based on respiratory signals depend on the signals considered and feature extraction methods. Moreover, feature engineering techniques are highly dependent on the experts’ experience and their prior knowledge about different physiological signals and conditions of the subjects. To overcome these problems, a novel deep recurrent neural network (RNN) framework is developed for automated feature extraction and detection of apnea events from single respiratory channel inputs. Long short-term memory (LSTM) and bidirectional long short-term memory (BiLSTM) are investigated to develop the proposed deep RNN model. The proposed framework is evaluated over three respiration signals: Oronasal thermal airflow (FlowTh), nasal pressure (NPRE), and abdominal respiratory inductance plethysmography (ABD). To demonstrate our results, we use polysomnogra...

Many of the recent advances of neural networks in sequential tasks such as natural language processing applications hinge on the use of representations obtained by predictive models. This success is usually ascribed to the emergence of neural representations that capture the low-dimensional latent structure implicit in the task. Motivated by the recent theoretical proposal that the hippocampus performs its role in sequential planning by organizing semantically related episodes in a relational network, we investigate the hypothesis that this organization results from learning a predictive representation of the world. Using an artificial recurrent neural network model trained with predictive learning on a simulated spatial navigation task, we show that network dynamics exhibit low dimensional but non-linearly transformed representations of sensory input statistics. These neural activations that are strongly reminiscent of the place-related neural activity that is experimentally observ...

Nature Communications

Artificial neural networks have recently achieved many successes in solving sequential processing and planning tasks. Their success is often ascribed to the emergence of the task’s low-dimensional latent structure in the network activity – i.e., in the learned neural representations. Here, we investigate the hypothesis that a means for generating representations with easily accessed low-dimensional latent structure, possibly reflecting an underlying semantic organization, is through learning to predict observations about the world. Specifically, we ask whether and when network mechanisms for sensory prediction coincide with those for extracting the underlying latent variables. Using a recurrent neural network model trained to predict a sequence of observations we show that network dynamics exhibit low-dimensional but nonlinearly transformed representations of sensory inputs that map the latent structure of the sensory environment. We quantify these results using nonlinear measures o...

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, 2017

Due to its variability, the development of wind power entails several difficulties, including wind speed forecasting. The Long Short-Term Memory (LSTM) is a particular type of recurrent network that can be used to work with sequential data, and previous works showed good empirical results. However, its training algorithm is expensive in terms of computation time. This paper proposes an efficient algorithm to train LSTM, decreasing computation time while maintaining good performance. The proposal is organized in two stages: (i) first to improve the weights output layer; (ii) next, update all weights using the original algorithm with one epoch. We used the proposed method to forecast wind speeds from 1 to 24 h ahead. Results demonstrated that our algorithm outperforms the original training algorithm, improving the efficiency and achieving better or comparable performance in terms of MAPE, MAE and RMSE.

Energies

Wind power generation has presented an important development around the world. However, its integration into electrical systems presents numerous challenges due to the variable nature of the wind. Therefore, to maintain an economical and reliable electricity supply, it is necessary to accurately predict wind generation. The Wind Power Prediction Tool (WPPT) has been proposed to solve this task using the power curve associated with a wind farm. Recurrent Neural Networks (RNNs) model complex non-linear relationships without requiring explicit mathematical expressions that relate the variables involved. In particular, two types of RNN, Long Short-Term Memory (LSTM) and Echo State Network (ESN), have shown good results in time series forecasting. In this work, we present an LSTM+ESN architecture that combines the characteristics of both networks. An architecture similar to an ESN is proposed, but using LSTM blocks as units in the hidden layer. The training process of this network has two key stages: (i) the hidden layer is trained with a descending gradient method online using one epoch; (ii) the output layer is adjusted with a regularized regression. In particular, the case is proposed where Step (i) is used as a target for the input signal, in order to extract characteristics automatically as the autoencoder approach; and in the second stage (ii), a quantile regression is used in order to obtain a robust estimate of the expected target. The experimental results show that LSTM+ESN using the autoencoder and quantile regression outperforms the WPPT model in all global metrics used.

Sensors

Sustainable freshwater management is underpinned by technologies which improve the efficiency of agricultural irrigation systems. Irrigation scheduling has the potential to incorporate real-time feedback from soil moisture and climatic sensors. However, for robust closed-loop decision support, models of the soil moisture dynamics are essential in order to predict crop water needs while adapting to external perturbation and disturbances. This paper presents a Dynamic Neural Network approach for modelling of the temporal soil moisture fluxes. The models are trained to generate a one-day-ahead prediction of the volumetric soil moisture content based on past soil moisture, precipitation, and climatic measurements. Using field data from three sites, a R 2 value above 0.94 was obtained during model evaluation in all sites. The models were also able to generate robust soil moisture predictions for independent sites which were not used in training the models. The application of the Dynamic ...

Processes

Clinical trials of single drugs intended to slow the progression of Alzheimer’s Disease (AD) have been notoriously unsuccessful. Combinations of repurposed drugs could provide effective treatments for AD. The challenge is to identify potentially effective combinations. To meet this challenge, machine learning (ML) was used to extract the knowledge from two leading AD databases, and then “the machine” predicted which combinations of the drugs in common between the two databases would be the most effective as treatments for AD. Specifically, three-layered artificial neural networks (ANNs) with compound, gated units in their internal layer were trained using ML to predict the cognitive scores of participants, separately in either database, given other data fields including age, demographic variables, comorbidities, and drugs taken. The predictions from the separately trained ANNs were statistically highly significantly correlated. The best drug combinations, jointly determined from bot...

Sensors

The analysis of data from sensors in structures subjected to extreme conditions such as the ones used in smelting processes is a great decision tool that allows knowing the behavior of the structure under different operational conditions. In this industry, the furnaces and the different elements are fully instrumented, including sensors to measure variables such as temperature, pressure, level, flow, power, electrode positions, among others. From the point of view of engineering and data analytics, this quantity of data presents an opportunity to understand the operation of the system under normal conditions or to explore new ways of operation by using information from models provided by using deep learning approaches. Although some approaches have been developed with application to this industry, it is still an open research area. As a contribution, this paper presents an applied deep learning temperature prediction model for a 75 MW electric arc furnace, which is used for ferronic...

2017

The paper aims at synthesis of an adaptive controller of the distillate output flow rate of a binary distillation column. The disturbance of the process is the change of concentration of the inlet compound. The Adaptive Critic Design (ACD) approach was applied to predict on time the future effect of disturbance and to adapt the distillate output flow rate in order to prevent deviations from the desired distillate concentration. The key element of ACD — the critic — is a fast trainable recurrent neural network named Echo state network (ESN). The simulation investigations demonstrated that the proposed adaptive control scheme outperforms a classical non-adaptive controller with respect to the settling time and the reaction delay.

Multimedia Tools and Applications

Medical imaging plays a significant role in different clinical applications such as medical procedures used for early detection, monitoring, diagnosis, and treatment evaluation of various medical conditions. Basicsof the principles and implementations of artificial neural networks and deep learning are essential for understanding medical image analysis in computer vision. Deep Learning Approach (DLA) in medical image analysis emerges as a fast-growing research field. DLA has been widely used in medical imaging to detect the presence or absence of the disease. This paper presents the development of artificial neural networks, comprehensive analysis of DLA, which delivers promising medical imaging applications. Most of the DLA implementations concentrate on the X-ray images, computerized tomography, mammography images, and digital histopathology images. It provides a systematic review of the articles for classification, detection, and segmentation of medical images based on DLA. This review guides the researchers to think of appropriate changes in medical image analysis based on DLA.

Service Oriented Computing and Applications

Supervised deep learning has achieved remarkable success in various applications. Successful machine learning application however depends on the availability of sufficiently large amount of data. In the absence of data from the target domain, representative data collection from multiple sources is often needed. However, a model trained on existing multi-source data might generalize poorly on the unseen target domain. This problem is referred to as domain shift. In this paper, we explore the suitability of multi-source training data selection to tackle the domain shift challenge in the context of domain generalization. We also propose a microservice-oriented methodology for supporting this solution. We perform our experimental study on the use case of building energy consumption prediction. Experimental results suggest that minimal building description is capable of improving cross-building generalization performances when used to select energy consumption data.

EPJ Web of Conferences

The safe operation of nuclear power plants is highly dependent on the ability of quickly and accurately identifying possible anomalies and perturbations in the reactor. Operational defects are primarily diagnosed by detectors that capture changes in the neutron flux, placed at various points inside and outside of the core. Neutron flux signals are subsequently analyzed with signal processing techniques in an effort to be better described (have their higher-order characteristics uncovered, locate transient events, etc). To this end, the application of intelligent techniques may be extremely beneficial, as it may assist and extend the current level of analysis. Besides, the combination of signal processing methodologies and machine learning techniques in the framework of nuclear power plant data is an emerging topic that has yet to show its full potential. In this context, the current contribution attempts at introducing intelligent approaches and more specifically, deep learning tech...

Applied Sciences

Extracting cybersecurity entities and the relationships between them from online textual resources such as articles, bulletins, and blogs and converting these resources into more structured and formal representations has important applications in cybersecurity research and is valuable for professional practitioners. Previous works to accomplish this task were mainly based on utilizing feature-based models. Feature-based models are time-consuming and need labor-intensive feature engineering to describe the properties of entities, domain knowledge, entity context, and linguistic characteristics. Therefore, to alleviate the need for feature engineering, we propose the usage of neural network models, specifically the long short-term memory (LSTM) models to accomplish the tasks of Named Entity Recognition (NER) and Relation Extraction (RE). We evaluated the proposed models on two tasks. The first task is performing NER and evaluating the results against the state-of-the-art Conditional R...

Applications of Evolutionary Computation

This paper presents the Evolutionary eXploration of Augmenting LSTM Topologies (EXALT) algorithm and its use in evolving recurrent neural networks (RNNs) for time series data prediction. It introduces a new open data set from a coal-fired power plant, consisting of 10 days of per minute sensor recordings from 12 different burners at the plant. This large scale real world data set involves complex depedencies between sensor parameters and makes for challening data to predict. EXALT provides interesting new techniques for evolving neural networks, including epigenetic weight initialization, where child neural networks re-use parental weights as a starting point to backpropagation, as well as node-level mutation operations which can improve evolutionary progress. EXALT has been designed with parallel computation in mind to further improve performance. Preliminary results were gathered predicting the Main Flame Intensity data parameter, with EXALT strongly outperforming five traditional neural network architectures on the best, average and worst cases across 10 repeated training runs per test case; and was only slightly behind the best trained Elman recurrent neural networks while being significantly more reliable (i.e., much better average and worst case results). Further, EXALT achived these results 2 to 10 times faster than the traditional methods, in part due to its scalability, showing strong potential to beat traditional architectures given additional runtime.

Proceedings of the Genetic and Evolutionary Computation Conference

This work examines the use of ant colony optimization (ACO) to improve long short-term memory (LSTM) recurrent neural networks (RNNs) by refining their cellular structure. The evolved networks were trained on a large database of flight data records obtained from an airline containing flights that suffered from excessive vibration. Results were obtained using MPI (Message Passing Interface) on a high performance computing (HPC) cluster, which evolved 1000 different LSTM cell structures using 208 cores over 5 days. The new evolved LSTM cells showed an improvement in prediction accuracy of 1.37%, reducing the mean prediction error from 6.38% to 5.01% when predicting excessive engine vibrations 10 seconds in the future, while at the same time dramatically reducing the number of trainable weights from 21,170 to 11,650. The ACO optimized LSTM also performed significantly better than traditional Nonlinear Output Error (NOE), Nonlinear AutoRegression with eXogenous (NARX) inputs, and Nonlinear Box-Jenkins (NBJ) models, which only reached error rates of 11.45%, 8.47% and 9.77%, respectively. The ACO algorithm employed could be utilized to optimize LSTM RNNs for any time series data prediction task.

Proceedings of the Genetic and Evolutionary Computation Conference

This paper presents a new algorithm, Evolutionary eXploration of Augmenting Memory Models (EXAMM), which is capable of evolving recurrent neural networks (RNNs) using a wide variety of memory structures, such as ∆-RNN, GRU, LSTM, MGU and UGRNN cells. EXAMM evolved RNNs to perform prediction of large-scale, real world time series data from the aviation and power industries. These data sets consist of very long time series (thousands of readings), each with a large number of potentially correlated and dependent parameters. Four different parameters were selected for prediction and EXAMM runs were performed using each memory cell type alone, each cell type and simple neurons, and with all possible memory cell types and simple neurons. Evolved RNN performance was measured using repeated k-fold cross validation, resulting in 2420 EXAMM runs which evolved 4, 840, 000 RNNs in ∼24, 200 CPU hours on a high performance computing cluster. Generalization of the evolved RNNs was examined statistically, providing interesting findings that can help refine the RNN memory cell design as well as inform future neuro-evolution algorithms development.

2020

Neuro-evolution and neural architecture search algorithms have gained significant interest due to the challenges of designing optimal artificial neural networks (ANNs). While these algorithms possess the potential to outperform the best human crafted architectures, a less common use of them is as a tool for analysis of ANN topologies and structural components. By performing these techniques while varying the allowable components, the best performing architectures for those components can be found and compared to best performing architectures for other components, allowing for a best case comparison of component capabilities – a more rigorous examination than simply applying those components in some standard fixed topologies. In this work, we utilize the Evolutionary eXploration of Augmenting Memory Models (EXAMM) algorithm to perform a rigorous examination and comparison of recurrent neural networks (RNNs) applied to time series prediction. Specifically, EXAMM is used to investigate...

2021

Neuroevolution commonly uses speciation strategies to better explore the search space of neural network architectures. One such speciation strategy is the use of islands, which are also popular in improving the performance of distributed evolutionary algorithms. However, islands may experience stagnation, which prevents their convergence towards better solutions and can result in wasted computation. This work evaluates utilizing an island extinction and repopulation mechanism to avoid premature convergence using Evolutionary eXploration of Augmenting Memory Models (EXAMM), an asynchronous island based neuroevolution algorithm that progressively evolves recurrent neural networks (RNNs). In island extinction and repopulation, all members of the worst performing island are erased periodically and repopulated with mutated versions of the global best RNN. This island based strategy is additionally compared to NEAT’s (NeuroEvolution of Augmenting Topologies) speciation strategy. Experimen...

2020

Transfer learning involves taking an artificial neural network (ANN) trained on one dataset (the source) and adapting it to a new, second dataset (the target). While transfer learning has been shown to be quite powerful and is commonly used in most modern-day statistical learning setups, its use has generally been restricted by architecture, i.e., in order to facilitate the reuse of internal learned synaptic weights, the underlying topology of the ANN to be transferred across tasks must remain the same and a new output layer must be attached (entailing removing the old output layer’s weights). This work removes this restriction by proposing a neuro-evolutionary approach that facilitates what we call adaptive structure transfer learning, which means that an ANN can be transferred across tasks that have different input and output dimensions while having the internal latent structure continuously optimized. We test the proposed optimizer on two challenging real-world time series predic...

Nature Communications, 2021

We propose a Double EXponential Adaptive Threshold (DEXAT) neuron model that improves the performance of neuromorphic Recurrent Spiking Neural Networks (RSNNs) by providing faster convergence, higher accuracy and a flexible long short-term memory. We present a hardware efficient methodology to realize the DEXAT neurons using tightly coupled circuit-device interactions and experimentally demonstrate the DEXAT neuron block using oxide based non-filamentary resistive switching devices. Using experimentally extracted parameters we simulate a full RSNN that achieves a classification accuracy of 96.1% on SMNIST dataset and 91% on Google Speech Commands (GSC) dataset. We also demonstrate full end-to-end real-time inference for speech recognition using real fabricated resistive memory circuit based DEXAT neurons. Finally, we investigate the impact of nanodevice variability and endurance illustrating the robustness of DEXAT based RSNNs.

2019

One of the major challenges of computational cognitive neuroscience is to apply models of neural information processing to complex tasks. Hierarchical learning architectures like deep convolutional networks can be trained to solve tasks efficiently, but utilize simple mechanisms of activity integration and output generation. On the other hand, biologically plausible models of activation dynamics incorporate detailed mechanisms of changing membrane potentials and axonal firing properties. Making such elaborate models trainable requires learning of internal model parameters. Here, we propose to apply supervised learning and train a model of canonical cortical circuits via backpropagation through time. We train the model to settle to target equilibrium values, to generate oscillations, and to solve a contour completion task.

PLOS Computational Biology

Place cells in the mammalian hippocampus signal self-location with sparse spatially stable firing fields. Based on observation of place cell activity it is possible to accurately decode an animal's location. The precision of this decoding sets a lower bound for the amount of information that the hippocampal population conveys about the location of the animal. In this work we use a novel recurrent neural network (RNN) decoder to infer the location of freely moving rats from single unit hippocampal recordings. RNNs are biologically plausible models of neural circuits that learn to incorporate relevant temporal context without the need to make complicated assumptions about the use of prior information to predict the current state. When decoding animal position from spike counts in 1D and 2D-environments, we show that the RNN consistently outperforms a standard Bayesian approach with either flat priors or with memory. In addition, we also conducted a set of sensitivity analysis on the RNN decoder to determine which neurons and sections of firing fields were the most influential. We found that the application of RNNs to neural data allowed flexible integration of temporal context, yielding improved accuracy relative to the more commonly used Bayesian approaches and opens new avenues for exploration of the neural code.

Sustainability, 2021

Hydro-meteorological datasets are key components for understanding physical hydrological processes, but the scarcity of observational data hinders their potential application in poorly gauged regions. Satellite-retrieved and atmospheric reanalysis products exhibit considerable advantages in filling the spatial gaps in in-situ gauging networks and are thus forced to drive the physically lumped hydrological models for long-term streamflow simulation in data-sparse regions. As machine learning (ML)-based techniques can capture the relationship between different elements, they may have potential in further exploring meteorological predictors and hydrological responses. To examine the application prospects of a physically constrained ML algorithm using earth observation data, we used a short-series hydrological observation of the Hanjiang River basin in China as a case study. In this study, the prevalent modèle du Génie Rural à 9 paramètres Journalier (GR4J-9) hydrological model was used...

Neural Computing and Applications

In this paper, we investigate the parameter identification problem in dynamical systems through a deep learning approach. Focusing mainly on second-order, linear time-invariant dynamical systems, the topic of damping factor identification is studied. By utilizing a six-layer deep neural network with different recurrent cells, namely GRUs, LSTMs or BiLSTMs; and by feeding input/output sequence pairs captured from a dynamical system simulator, we search for an effective deep recurrent architecture in order to resolve damping factor identification problem. Our study's results show that, although previously not utilized for this task in the literature, bidirectional gated recurrent cells (BiLSTMs) provide better parameter identification results when compared to unidirectional gated recurrent memory cells such as GRUs and LSTM. Thus, indicating that an input/output sequence pair of finite length, collected from a dynamical system and when observed anachronistically, may carry information in both time directions for prediction of a dynamical systems parameter.

2021

When using LSTM networks to model time-series data, the standard approach is to segment the continuous data stream into fixedsize sequences and then independently feed each sequence to the LSTM network for training in a stateless fashion (i.e in a fashion that resets the LSTM cell state per fixed-size sequence). As a result, long-term dependencies between patterns appearing in the data stream may be lost. In this work, we introduce a hybrid deep learning architecture that enables long-term inter-sequence modeling while maintaining focus on each sequence’s local characteristics. We use stateful LSTM training to model long-term dependencies that span the fixed-size sequences. We also utilize the attention mechanism to optimally learn each training sequence by focusing on the parts of each sequence that affect the classification outcome the most. Our experimental results show the advantages of each of these two mechanisms independently and in conjunction, compared to the standard state...

Journal of Artificial Intelligence and Soft Computing Research, 2021

Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and time-series predictions. Such networks are usually trained through back-propagation through time (BPTT) which is prohibitively expensive, especially when the length of the time dependencies and the number of hidden neurons increase. To reduce the training time, extreme learning machines (ELMs) have been recently applied to RNN training, reaching a 99% speedup on some applications. Due to its non-iterative nature, ELM training, when parallelized, has the potential to reach higher speedups than BPTT. In this work, we present Opt-PR-ELM, an optimized parallel RNN training algorithm based on ELM that takes advantage of the GPU shared memory and of parallel QR factorization algorithms to efficiently reach optimal solutions. The theoretical analysis of the proposed algorithm is presented on six RNN architectures, including LSTM and GRU, ...

GEM - International Journal on Geomathematics

This work presents a hybrid modeling approach to data-driven learning and representation of unknown physical processes and closure parameterizations. These hybrid models are suitable for situations where the mechanistic description of dynamics of some variables is unknown, but reasonably accurate observational data can be obtained for the evolution of the state of the system. In this work, we propose machine learning to account for missing physics and then data assimilation to correct the prediction. In particular, we devise an effective methodology based on a recurrent neural network to model the unknown dynamics. A long short-term memory (LSTM) based correction term is added to the predictive model in order to take into account hidden physics. Since LSTM introduces a black-box approach for the unknown part of the model, we investigate whether the proposed hybrid neural-physical model can be further corrected through a sequential data assimilation step. We apply this framework to the weakly nonlinear Lorenz model that displays quasiperiodic oscillations, the highly nonlinear chaotic Lorenz model, and two-scale Lorenz model. The hybrid neural-physics model yields accurate results for the weakly nonlinear Lorenz model with the predicted state close to the true Lorenz model trajectory. For the highly nonlinear Lorenz model and the two-scale Lorenz model, the hybrid neural-physics model deviates from the true state due to the accumulation of prediction error from one time step to the next time step. The ensemble Kalman filter approach takes into account the prediction error and updates the diverged prediction using available observations in order to provide a more accurate state estimate for the highly nonlinear chaotic Lorenz model and two-scale Lorenz system. The successful synergistic integration of neural network and data assimilation for low-dimensional system shows the potential benefits of the proposed hybrid-neural physics model for complex dynamical systems.

Computer Vision – ECCV 2020

In the recent quest for trustworthy neural networks, we present Spiking Neural Network (SNN) as a potential candidate for inherent robustness against adversarial attacks. In this work, we demonstrate that accuracy degradation is less severe in SNNs than in their non-spiking counterparts for CIFAR10 and CIFAR100 datasets on deep VGG architectures. We attribute this robustness to two fundamental characteristics of SNNs and analyze their effects. First, we exhibit that input discretization introduced by the Poisson encoder improves adversarial robustness with reduced number of timesteps. Second, we quantify the amount of adversarial accuracy with increased leak rate in Leaky-Integrate-Fire (LIF) neurons. Our results suggest that SNNs trained with LIF neurons and smaller number of timesteps are more robust than the ones with IF (Integrate-Fire) neurons and larger number of timesteps. We overcome the bottleneck of creating gradient-based adversarial inputs in temporal domain by proposing a technique for crafting attacks from SNN.

International Journal of Data Science and Analytics, 2017

Today, there are a large number of online discussion fora on the internet which are meant for users to express, discuss and exchange their views and opinions on various topics. For example, news portals, blogs, social media channels such as youtube. typically allow users to express their views through comments. In such fora, it has been often observed that user conversations sometimes quickly derail and become inappropriate such as hurling abuses, passing rude and discourteous comments on individuals or certain groups/communities. Similarly, some virtual agents or bots have also been found to respond back to users with inappropriate messages. As a result, inappropriate messages or comments are turning into an online menace slowly degrading the effectiveness of user experiences. Hence, automatic detection and filtering of such inappropriate language has become an important problem for improving the quality of conversations with users as well as virtual agents. In this paper, we propo...

ABSTRACTSequential behaviour is often compositional and organised across multiple time scales: a set of individual elements developing on short time scales (motifs) are combined to form longer functional sequences (syntax). Such organisation leads to a natural hierarchy that can be used advantageously for learning, since the motifs and the syntax can be acquired independently. Despite mounting experimental evidence for hierarchical structures in neuroscience, models for temporal learning based on neuronal networks have mostly focused on serial methods. Here, we introduce a network model of spiking neurons with a hierarchical organisation aimed at sequence learning on multiple time scales. Using biophysically motivated neuron dynamics and local plasticity rules, the model can learn motifs and syntax independently. Furthermore, the model can relearn sequences efficiently and store multiple sequences. Compared to serial learning, the hierarchical model displays faster learning, more fl...

Biotechnology & Biotechnological Equipment, 1994

All modern control theories require a sufficiently accurate model of the plant to be controlled. In the case of complex nonlinear models an alternative to identify the dynamics of the system is to use neural network methodology because there is no need to know in advance functional form of the model-it is determined simultaneously with model parameters. In the present paper two neutral network models of S. carsbergensis batch cultivation are proposed. Fmm the obtained res•ults il can be concluded that neural networks allow to r•each higher accuracy of non-linear models. The possibility of adding internal memory in the model without raising the cost of identification makes neural networks very appropriate for• modelling such ~om.plex non-linear systems as biotechnological processes.

7th AIAA ATIO Conf, 2nd CEIAT Int'l Conf on Innov and Integr in Aero Sciences,17th LTA Systems Tech Conf; followed by 2nd TEOS Forum, 2007

The failure rate analysis of brake assemblies of a commercial airplane, i.e., Boeing 737, is analyzed using the artificial neural network and Weibull regression models. One-layered feed-forward backpropagation algorithm for artificial neural network whereas three parameters model for Weibull are used for the analysis. Three years of data are used for model building and validation. The results show that the failure rate predicted by neural network is closer in agreement with the actual data than the failure rate predicted by the Weibull model. Results also indicate that neural network can be effectively integrated into an aviation maintenance facility computerized material requirement planning system to forecast the number of brake assemblies needed for a given planning horizon.

Computational Science and Its Applications – ICCSA 2019

A direct impact of population density is more cities suffering from constant traffic jams. Thinking this way, Intelligent Transportation Systems, a key area in smart cities, uses computational intelligence techniques and analyses to aid in traffic dimensioning solutions. In this context, accurate traffic prediction models are vital to creating a more autonomous and intelligent environment. With an increase in projects for intelligent cities, research in the area of computational intelligence becomes a necessity, since its models can address complex real-world problems, which are usually difficult for conventional methods. In this work, an application is introduced applying machine learning to empower a smart ecosystem. To validate it, an extensive evaluation was performed, comparing it with the state-of-the-art and, also, verifying the impact of parameter variation and activation functions on the model of traffic flow prediction. All evaluations were done using real data traffic of two very distinct scenarios. Firstly, a free traffic flow scenario was evaluated in a benchmark dataset. Then, both models were evaluated in a complex traffic scenario where traffic flow is not continuous nor large. In both scenarios, the presented application, called SmartTraffic, outperforms the current state-of-the-art, with a performance gain of over 100% when compared in the first scenario and an improvement of approximately 31%, on average, in the second one.

Royal Society Open Science

The use of machine learning has grown in popularity in various disciplines. Despite the popularity, the apparent ‘black box’ nature of such tools continues to be an area of concern. In this article, we attempt to unravel the complexity of this black box by exploring the use of artificial neural networks (ANNs), coupled with graph theory, to model and interpret the spatial distribution of building damage from extreme wind events at a community level. Structural wind damage is a topic that is mostly well understood for how wind pressure translates to extreme loading on a structure, how debris can affect that loading and how specific social characteristics contribute to the overall population vulnerability. While these themes are widely accepted, they have proven difficult to model in a cohesive manner, which has led primarily to physical damage models considering wind loading only as it relates to structural capacity. We take advantage of this modelling difficulty to reflect on two di...

Electronics

Nowadays, the information provided by digital photographs is very complete and very relevant in different professional fields, such as scientific or forensic photography. Taking this into account, it is possible to determine the date when they were taken, as well as the type of device that they were taken with, and thus be able to locate the photograph in a specific context. This is not the case with analog photographs, which lack any information regarding the date they were taken. Extracting this information is a complicated task, so classifying each photograph according to the date it was taken is a laborious task for a human expert. Artificial intelligence techniques make it possible to determine the characteristics and classify the images automatically. Within the field of artificial intelligence, convolutional neural networks are one of the most widely used methods today. This article describes the application of convolutional neural networks to automatically classify photograp...

Machine Learning and Knowledge Discovery in Databases, 2020

In neural networks, the learning rate of the gradient descent strongly affects performance. This prevents reliable out-of-the-box training of a model on a new problem. We propose the All Learning Rates At Once (Alrao) algorithm: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude, in the hope that enough units will get a close-to-optimal learning rate. Perhaps surprisingly, stochastic gradient descent (SGD) with Alrao performs close to SGD with an optimally tuned learning rate, for various network architectures and problems. In our experiments, all Alrao runs were able to learn well without any tuning.

ACM Transactions on Intelligent Systems and Technology

Short text is ambiguous and often relies predominantly on the domain and context at hand in order to attain semantic relevance. Existing classification models perform poorly on short text due to data sparsity and inadequate context. Auxiliary context, which can often provide sufficient background regarding the domain, is typically available in several application scenarios. While some of the existing works aim to leverage real-world knowledge to enhance short-text representations, they fail to place appropriate emphasis on the auxiliary context. Such models do not harness the full potential of the available context in auxiliary sources. To address this challenge, we reformulate short-text classification as a dual channel self-supervised learning problem (that leverages auxiliary context) with a generation network and a corresponding prediction model. We propose a self-supervised framework, Pseudo-Auxiliary Context generation network for Short-text Modeling (PACS) , to comprehensivel...

2017

ABSTRACTThe human visual cortex extracts both spatial and temporal visual features to support perception and guide behavior. Deep convolutional neural networks (CNNs) provide a computational framework to model cortical representation and organization for spatial visual processing, but unable to explain how the brain processes temporal information. To overcome this limitation, we extended a CNN by adding recurrent connections to different layers of the CNN to allow spatial representations to be remembered and accumulated over time. The extended model, or the recurrent neural network (RNN), embodied a hierarchical and distributed model of process memory as an integral part of visual processing. Unlike the CNN, the RNN learned spatiotemporal features from videos to enable action recognition. The RNN better predicted cortical responses to natural movie stimuli than the CNN, at all visual areas especially those along the dorsal stream. As a fully-observable model of visual processing, th...

Transport in Porous Media, 2018

We present a reduced-order modeling technique for subsurface multi-phase flow problems building on the recently introduced deep residual recurrent neural network (DR-RNN) (Nagoor Kani et al. in DR-RNN: a deep residual recurrent neural network for model reduction. ArXiv e-prints, 2017). DR-RNN is a physics-aware recurrent neural network for modeling the evolution of dynamical systems. The DR-RNN architecture is inspired by iterative update techniques of line search methods where a fixed number of layers are stacked together to minimize the residual (or reduced residual) of the physical model under consideration. In this manuscript, we combine DR-RNN with proper orthogonal decomposition (POD) and discrete empirical interpolation method (DEIM) to reduce the computational complexity associated with high-fidelity numerical simulations. In the presented formulation, POD is used to construct an optimal set of reduced basis functions and DEIM is employed to evaluate the nonlinear terms independent of the full-order model size. We demonstrate the proposed reduced model on two uncertainty quantification test cases using Monte Carlo simulation of subsurface flow with random permeability field. The obtained results demonstrate that DR-RNN combined with POD-DEIM provides an accurate and stable reduced model with a fixed computational budget that is much less than the computational cost of standard POD-Galerkin reduced model combined with DEIM for nonlinear dynamical systems.

Applied Sciences, 2017

The surface finishing and stock removal of complicated geometries is the principal objective for grinding with compliant abrasive tools. To understand and achieve optimum material removal in a tertiary finishing process such as Abrasive Belt Grinding, it is essential to look in more detail at the process parameters/variables that affect the stock removal rate. The process variables involved in a belt grinding process include the grit and abrasive type of grinding belt, belt speed, contact wheel hardness, serration, and grinding force. Changing these process variables will affect the performance of the process. The literature survey on belt grinding shows certain limited understanding of material removal on the process variables. Experimental trials were conducted based on the Taguchi Method to evaluate the influence of individual and interactive process variables. Analysis of variance (ANOVA) was employed to investigate the belt grinding characteristics on material removal. This research work describes a systematic approach to optimise process parameters to achieve the desired stock removal in a compliant Abrasive Belt Grinding process. Experimental study showed that the removed material from a surface due to the belt grinding process has a non-linear relationship with the process variables. In this paper, the Adaptive Neuro-Fuzzy Inference System (ANFIS) model is used to determine material removal. Compared with the experimental results, the model accurately predicts the stock removal. With further verification of the empirical model, a better understanding of the grinding parameters involved in material removal, particularly the influence of the individual process variables and their interaction, can be obtained.

Artificial Life, 2013

Anthropomimetic robotics differs from conventional approaches by capitalizing on the replication of the inner structures of the human body, such as muscles, tendons, bones, and joints. Here we present our results of more than three years of research in constructing, simulating, and, most importantly, controlling anthropomimetic robots. We manufactured four physical torsos, each more complex than its predecessor, and developed the tools required to simulate their behavior. Furthermore, six different control approaches, inspired by classical control theory, machine learning, and neuroscience, were developed and evaluated via these simulations or in small-scale setups. While the obtained results are encouraging, we are aware that we have barely exploited the potential of the anthropomimetic design so far. But, with the tools developed, we are confident that this novel approach will contribute to our understanding of morphological computation and human motor control in the future.

Scientific Reports, 2021

Physical reservoir computing utilizes a physical system as a computational resource. This nontraditional computing technique can be computationally powerful, without the need of costly training. Here, a Hopf oscillator is implemented as a reservoir computer by using a node-based architecture; however, this implementation does not use delayed feedback lines. This reservoir computer is still powerful, but it is considerably simpler and cheaper to implement as a physical Hopf oscillator. A non-periodic stochastic masking procedure is applied for this reservoir computer following the time multiplexing method. Due to the presence of noise, the Euler–Maruyama method is used to simulate the resulting stochastic differential equations that represent this reservoir computer. An analog electrical circuit is built to implement this Hopf oscillator reservoir computer experimentally. The information processing capability was tested numerically and experimentally by performing logical tasks, emul...

International Journal of Intelligent Robotics and Applications, 2019

Functional electrical stimulation (FES) has recently been proposed as a supplementary torque assist in lower-limb powered exoskeletons for persons with paraplegia. In the combined system, also known as a hybrid neuroprosthesis, both FES-assist and the exoskeleton act to generate lower-limb torques to achieve standing and walking functions. Due to this actuator redundancy, we are motivated to optimally allocate FES-assist and exoskeleton torque based on a performance index that penalizes FES overuse to minimize muscle fatigue while also minimizing regulation or tracking errors. Traditional optimal control approaches need a system model to optimize; however, it is often difficult to formulate a musculoskeletal model that accurately predicts muscle responses due to FES. In this paper, we use a novel identification and control structure that contains a recurrent neural network (RNN) and several feedforward neural networks (FNNs). The RNN is trained by supervised learning to identify the system dynamics, while the FNNs are trained by a reinforcement learning method to provide suboptimal control actions. The output layer of each FNN has its unique activation functions, so that the asymmetric constraint of FES and the symmetric constraint of exoskeleton motor control input can be realized. This new structure is experimentally validated on a seated human participant using a single joint hybrid neuroprosthesis.

Neurocomputing, 2021

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Computational Geosciences, 2020

Borehole resistivity measurements are routinely employed to measure the electrical properties of rocks penetrated by a well and to quantify the hydrocarbon pore volume of a reservoir. Depending on the degree of geometrical complexity, inversion techniques are often used to estimate layer-by-layer electrical properties from measurements. When used for well geosteering purposes, it becomes essential to invert the measurements into layer-by-layer values of electrical resistivity in real time. We explore the possibility of using deep neural networks (DNNs) to perform rapid inversion of borehole resistivity measurements. Accordingly, we construct a DNN that approximates the following inverse problem: given a set of borehole resistivity measurements, the DNN is designed to deliver a physically reliable and data-consistent piecewise one-dimensional layered model of the surrounding subsurface. Once the DNN is constructed, we can invert borehole measurements in real time. We illustrate the performance of the DNN for inverting logging-while-drilling (LWD) measurements acquired in highangle wells via synthetic examples. Numerical results are promising, although further work is needed to achieve the accuracy and reliability required by petrophysicists and drillers. Keywords Logging-while-drilling (LWD) • Resistivity measurements • Real-time inversion • Deep learning • Well geosteering • Deep neural networks

Smart Technologies, Systems and Applications, 2020

Language Identification (LID) is an essential research topic in the Automatic Recognition Speech area. One of the most important characteristics relative to language is context information. In this article, considering a phonotactic approach where the phonetic units called "phone-grams" are used, in order to introduce such context information, a novel technique is proposed. Language discriminative information has been incorporated in the Recurrent Neural Network Language Models generation (RNNLMs) in the weights initialization stage to improve the Language Identification task. This technique has been evaluated using KALAKA-3 database that contains 108 h of audios of six languages to be recognized. The metric used in this work has been the Average Detection Cost metric C avg. In relation to the phonetic units called "phone-grams" used in order to incorporate context information in the features used to train the RNNLM, it has been considered phone-grams of two elements "2phone-grams" and three elements "3phone-grams", obtaining a relative improvement up to 17% and 15,44% respectively compared to the results obtaining using RNNLMs.

Scientific Reports, 2020

Capsule endoscopy (CE) is a widely used, minimally invasive alternative to traditional endoscopy that allows visualisation of the entire small intestine. Patient preparation can help to obtain a cleaner intestine and thus better visibility in the resulting videos. However, studies on the most effective preparation method are conflicting due to the absence of objective, automatic cleanliness evaluation methods. In this work, we aim to provide such a method capable of presenting results on an intuitive scale, with a relatively light-weight novel convolutional neural network architecture at its core. We trained our model using 5-fold cross-validation on an extensive data set of over 50,000 image patches, collected from 35 different CE procedures, and compared it with state-of-the-art classification methods. From the patch classification results, we developed a method to automatically estimate pixel-level probabilities and deduce cleanliness evaluation scores through automatically learn...

Robotics and Autonomous Systems, 2021

Pouring is one of the most commonly executed tasks in humans' daily lives, whose accuracy is affected by multiple factors, including the type of material to be poured and the geometry of the source and receiving containers. In this work, we propose a self-supervised learning approach that learns the pouring dynamics, pouring motion, and outcomes from unsupervised demonstrations for accurate pouring. The learned pouring model is then generalized by self-supervised practicing to different conditions such as using unaccustomed pouring cups. We have evaluated the proposed approach first with one container from the training set and four new but similar containers. The proposed approach achieved better pouring accuracy than a regular human with a similar pouring speed for all five cups. Both the accuracy and pouring speed outperform state-of-the-art works. We have also evaluated the proposed self-supervised generalization approach using unaccustomed containers that are far different from the ones in the training set. The self-supervised generalization reduces the pouring error of the unaccustomed containers to the desired accuracy level.

Molecular Diversity, 2021

The introduction of a new drug to the commercial market follows a complex and long process that typically spans over several years and entails large monetary costs due to a high attrition rate. Because of this, there is an urgent need to improve this process using innovative technologies such as artificial intelligence (AI). Different AI tools are being applied to support all four steps of the drug development process (basic research for drug discovery; pre-clinical phase; clinical phase; and postmarketing). Some of the main tasks where AI has proven useful include identifying molecular targets, searching for hit and lead compounds, synthesising drug-like compounds and predicting ADME-Tox. This review, on the one hand, brings in a mathematical vision of some of the key AI methods used in drug development closer to medicinal chemists and, on the other hand, brings the drug development process and the use of different models closer to mathematicians. Emphasis is placed on two aspects ...

Applied Computational Intelligence and Soft Computing

Deep sequential (DS) models are extensively employed for forecasting time series data since the dawn of the deep learning era, and they provide forecasts for the values required in subsequent time steps. DS models, unlike other traditional statistical models for forecasting time series data, can learn hidden patterns in temporal sequences and have the memorizing data from prior time points. Given the widespread usage of deep sequential models in several domains, a comprehensive study describing their applications is necessary. This work presents a comprehensive review of contemporary deep learning time series models, their performance in diverse domains, and an investigation of the models that were employed in various applications. Three deep sequential models, namely, artificial neural network (ANN), long short-term memory (LSTM), and temporal-conventional neural network (TCNN) along with their applications for forecasting time series data, are elaborated. We showed a comprehensive...

Electronics

Continuous growth in software, hardware and internet technology has enabled the growth of internet-based sensor tools that provide physical world observations and data measurement. The Internet of Things(IoT) is made up of billions of smart things that communicate, extending the boundaries of physical and virtual entities of the world further. These intelligent things produce or collect massive data daily with a broad range of applications and fields. Analytics on these huge data is a critical tool for discovering new knowledge, foreseeing future knowledge and making control decisions that make IoT a worthy business paradigm and enhancing technology. Deep learning has been used in a variety of projects involving IoT and mobile apps, with encouraging early results. With its data-driven, anomaly-based methodology and capacity to detect developing, unexpected attacks, deep learning may deliver cutting-edge solutions for IoT intrusion detection. In this paper, the increased amount of in...

Journal of Open Source Education

Most fields of scientific inquiry require the evaluation of derivatives to calculate and optimize quantities of interest. Automatic differentiation is a set of techniques that allow the differentiation of computer programs to machine precision without requiring full symbolic derivatives (Baydin et al., 2017; Griewank, 1989). The great success of machine learning algorithms, and neural networks in particular, was partly enabled by the celebrated backpropagation algorithm (Werbos, 1990), which is a special case of automatic differentiation. Given the rapidly increasing interest in algorithms that rely on automatic differentiation, and the evolution towards differential programming paradigms (Innes et al., 2019), it is important that students be taught the basics of this key family of algorithms. Automatic differentiation is a method of computing derivatives to machine precision based on the decomposition of functions into a series of elementary operations. These operations can be conceptualized as forming a graph structure. This graph can be traversed in the forward or reverse direction, giving rise to the two primary modes in automatic differentiation. The goal of the Auto-eD software and the accompanying lecture modules is to enhance students' understanding of automatic differentiation by helping them to visualize the underlying graph structure of the computations.

Sensors

Bearings are vital components of rotating machines that are prone to unexpected faults. Therefore, bearing fault diagnosis and condition monitoring are essential for reducing operational costs and downtime in numerous industries. In various production conditions, bearings can be operated under a range of loads and speeds, which causes different vibration patterns associated with each fault type. Normal data are ample as systems usually work in desired conditions. On the other hand, fault data are rare, and in many conditions, there are no data recorded for the fault classes. Accessing fault data is crucial for developing data-driven fault diagnosis tools that can improve both the performance and safety of operations. To this end, a novel algorithm based on conditional generative adversarial networks (CGANs) was introduced. Trained on the normal and fault data on actual fault conditions, this algorithm generates fault data from normal data of target conditions. The proposed method wa...

Journal of New Music Research, 2017

In this paper, we propose a new system design framework for large vocabulary automatic chord estimation. Our approach is based on an integration of traditional sequence segmentation processes and deep learning chord classification techniques. We systematically explore the design space of the proposed framework for a range of parameters, namely deep neural nets, network configurations, input feature representations, segment tiling schemes, and training data sizes. Experimental results show that among the three proposed deep neural nets and a baseline model, the recurrent neural network based system has the best average chord quality accuracy that significantly outperforms the other considered models. Furthermore, our bias-variance analysis has identified a glass ceiling as a potential hindrance to future improvements of large vocabulary automatic chord estimation systems.

Information Retrieval Journal, 2017

In this paper we tackle the problem of image search when the query is a short textual description of the image the user is looking for. We choose to implement the actual search process as a similarity search in a visual feature space, by learning to translate a textual query into a visual representation. Searching in the visual feature space has the advantage that any update to the translation model does not require to reprocess the, typically huge, image collection on which the search is performed. We propose Text2Vis, a neural network that generates a visual representation, in the visual feature space of the fc6-fc7 layers of ImageNet, from a short descriptive text. Text2Vis optimizes two loss functions, using a stochastic loss-selection method. A visual-focused loss is aimed at learning the actual text-to-visual feature mapping, while a text-focused loss is aimed at modeling the higherlevel semantic concepts expressed in language and countering the overfit on non-relevant visual components of the visual loss. We report preliminary results on the MS-COCO dataset.

Physical Review Research, 2021

Despite the significance of short-term memory in cognitive function, the process of encoding and sustaining the input information in neural activity dynamics remains elusive. Herein, we unveiled the significance of transient neural dynamics to short-term memory. By training recurrent neural networks to short-term memory tasks and analyzing the dynamics, the characteristics of the shortterm memory mechanism were obtained in which the input information was encoded in the amplitude of transient oscillations, rather than the stationary neural activities. This transient trajectory was attracted to a slow manifold, which permitted the discarding of irrelevant information. Additionally, we investigated the process by which the dynamics acquire robustness to noise. In this transient oscillation, the robustness to noise was obtained by a strong contraction of the neural states after perturbation onto the manifold. This mechanism works for several neural network models and tasks, which implies its relevance to neural information processing in general.

Nature, 2016

Artificial neural networks are remarkably adept at sensory processing, sequence learning and reinforcement learning, but are limited in their ability to represent variables and data structures and to store data over long timescales, owing to the lack of an external memory. Here we introduce a machine learning model called a differentiable neural computer (DNC), which consists of a neural network that can read from and write to an external memory matrix, analogous to the random-access memory in a conventional computer. Like a conventional computer, it can use its memory to represent and manipulate complex data structures, but, like a neural network, it can learn to do so from data. When trained with supervised learning, we demonstrate that a DNC can successfully answer synthetic questions designed to emulate reasoning and inference problems in natural language. We show that it can learn tasks such as finding the shortest path between specified points and inferring the missing links in randomly generated graphs, and then generalize these tasks to specific graphs such as transport networks and family trees. When trained with reinforcement learning, a DNC can complete a moving blocks puzzle in which changing goals are specified by sequences of symbols. Taken together, our results demonstrate that DNCs have the capacity to solve complex, structured tasks that are inaccessible to neural networks without external read-write memory.

The flavor is an essential component in developing numerous products in the market. The increasing consumption of processed and fast food and healthy packages has upraised the investment in new flavoring agents and, consequently, molecules with flavoring properties. In this context, this work brings a Scientific Machine Learning approach to address this product engineering need. Scientific Machine Learning in computational chemistry has opened paths in predicting a compound's properties without requiring synthesis. This work proposes a novel framework of deep generative models within this context to design new flavor molecules.

Signals, 2022

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY

PLOS ONE, 2021

Social scientists and psychologists take interest in understanding how people express emotions and sentiments when dealing with catastrophic events such as natural disasters, political unrest, and terrorism. The COVID-19 pandemic is a catastrophic event that has raised a number of psychological issues such as depression given abrupt social changes and lack of employment. Advancements of deep learning-based language models have been promising for sentiment analysis with data from social networks such as Twitter. Given the situation with COVID-19 pandemic, different countries had different peaks where rise and fall of new cases affected lock-downs which directly affected the economy and employment. During the rise of COVID-19 cases with stricter lock-downs, people have been expressing their sentiments in social media. This can provide a deep understanding of human psychology during catastrophic events. In this paper, we present a framework that employs deep learning-based language mod...

Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2021

This work presents how the Evolutionary eXploration of Augmenting Memory Models (EXAMM) neuroevolution algorithm is incorporated into Microbeam Technologies' condition-based monitoring power plant optimization software using a workflow that integrates coal-fired power plant data collection, evolved RNN predictions and analytic performance indices predictions. To the authors' knowledge, it is the first use of a neuroevolution strategy to evolve recurrent neural networks (RNNs) for forecasting of power plant parameters where the evolved networks have been incorporated into production software used at a coal-fired power plant. A preliminary exploration of the plant's performance shows that after incorporating this software, the amount of revenue lost due to power plant derates and outages decreased by $7.3 million, a savings of 42%, and increased efficiency under medium and low load conditions. A further investigation of the effect of training sequence length and time series data normalization methods on evolving and training RNNs for this system is given, providing practical results useful for real world time series forecasting. It is shown that dividing long time series sequences up into shortened training sequences can dramatically speed up training, and that using different normalization methods (min-max vs. z-score) can provide statistically significant results, dependent on the data sets. CCS CONCEPTS • Computing methodologies → Neural networks; Genetic algorithms; • Applied computing → Forecasting.

H2Open Journal, 2021

ANN was used to create a storage-based concurrent flow forecasting model. River flow parameters in an unsteady flow must be modeled using a model formulation based on learning storage change variable and instantaneous storage rate change. Multiple input-multiple output (MIMO) and multiple input-single output (MISO models in three variants were used to anticipate flow rates in the Tar River Basin in the United States. Gamma memory neural networks, as well as MLP and TDNNs models, are used in this study. When issuing a forecast, storage variables for river flow must be considered, which is why this study includes them. While considering mass balance flow, the proposed model can provide real-time flow forecasting. Results obtained are validated using various statistical criteria such as RMS error and coefficient of correlation. For the models, a coefficient of correlation value of more than 0.96 indicates good results. While considering the mass balance flow, the results show flow fluc...

Informatica, 2020

Traffic flow forecasting is an acknowledged time series problem whose solutions have been essentially grounded on statistical-based models. Recent times came, however, with promising results regarding the use of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory networks (LSTMs), to accurately address time series problems. Literature is, however, evasive in regard to several aspects of the conceived models and often exhibits misconceptions that may lead to important pitfalls. This study aims to conceive and find the best possible LSTM model for traffic flow forecasting while addressing several important aspects of such models such as the multitude of input features, the time frames used by the model and the employed approach for multi-step forecasting. To overcome the spatial problem of open source datasets, this study presents and describes a new dataset collected by the authors of this work. After several weeks of model fitting, Recursive Multi-Step Multi-Variate models were the ones showing better performance, strengthening the perception that LSTMs can be used to accurately forecast the traffic flow for several future timesteps.

Journal of Engineering for Gas Turbines and Power, 2012

In diesel engines, variable geometry turbocharger (VGT) and exhaust gas recirculation (EGR) systems are used to increase engine specific power and reduce NOx emissions, respectively. Because the dynamics of both the VGT and EGR are highly nonlinear and coupled to each other, better performance may be attained by substituting nonlinear multiple input, multiple output (MIMO) controllers for the existing conventional lookup table-based linear controllers. This paper presents a coordinated VGT/EGR control system for common-rail direct injection diesel engines. The objective of the control system is to track target mass air flow and target intake manifold pressure by adjusting the EGR and VGT actuator positions. We designed a nonlinear MIMO control system using a neural control scheme that adopts an indirect adaptive control approach. The neural control system is comprised of a neural network identifier, which mimics the target air system, and a neural network controller, which calculate...

Frontiers in Neuroscience, 2018

In this work, we propose a Spiking Neural Network (SNN) consisting of input neurons sparsely connected by plastic synapses to a randomly interlinked liquid, referred to as Liquid-SNN, for unsupervised speech and image recognition. We adapt the strength of the synapses interconnecting the input and liquid using Spike Timing Dependent Plasticity (STDP), which enables the neurons to self-learn a general representation of unique classes of input patterns. The presented unsupervised learning methodology makes it possible to infer the class of a test input directly using the liquid neuronal spiking activity. This is in contrast to standard Liquid State Machines (LSMs) that have fixed synaptic connections between the input and liquid followed by a readout layer (trained in a supervised manner) to extract the liquid states and infer the class of the input patterns. Moreover, the utility of LSMs has primarily been demonstrated for speech recognition. We find that training such LSMs is challenging for complex pattern recognition tasks because of the information loss incurred by using fixed input to liquid synaptic connections. We show that our Liquid-SNN is capable of efficiently recognizing both speech and image patterns by learning the rich temporal information contained in the respective input patterns. However, the need to enlarge the liquid for improving the accuracy introduces scalability challenges and training inefficiencies. We propose SpiLinC that is composed of an ensemble of multiple liquids operating in parallel. We use a "divide and learn" strategy for SpiLinC, where each liquid is trained on a unique segment of the input patterns that causes the neurons to self-learn distinctive input features. SpiLinC effectively recognizes a test pattern by combining the spiking activity of the constituent liquids, each of which identifies characteristic input features. As a result, SpiLinC offers competitive classification accuracy compared to the Liquid-SNN with added sparsity in synaptic connectivity and faster training convergence, both of which lead to improved energy efficiency in neuromorphic hardware implementations. We validate the efficacy of the proposed Liquid-SNN and SpiLinC on the entire digit subset of the TI46 speech corpus and handwritten digits from the MNIST dataset.

Frontiers in Neuroscience, 2019

Spiking neural networks (SNNs) offer a promising alternative to current artificial neural networks to enable low-power event-driven neuromorphic hardware. Spike-based neuromorphic applications require processing and extracting meaningful information from spatio-temporal data, represented as series of spike trains over time. In this paper, we propose a method to synthesize images from multiple modalities in a spike-based environment. We use spiking auto-encoders to convert image and audio inputs into compact spatio-temporal representations that is then decoded for image synthesis. For this, we use a direct training algorithm that computes loss on the membrane potential of the output layer and back-propagates it by using a sigmoid approximation of the neuron's activation function to enable differentiability. The spiking autoencoders are benchmarked on MNIST and Fashion-MNIST and achieve very low reconstruction loss, comparable to ANNs. Then, spiking autoencoders are trained to learn meaningful spatio-temporal representations of the data, across the two modalities-audio and visual. We synthesize images from audio in a spike-based environment by first generating, and then utilizing such shared multi-modal spatio-temporal representations. Our audio to image synthesis model is tested on the task of converting TI-46 digits audio samples to MNIST images. We are able to synthesize images with high fidelity and the model achieves competitive performance against ANNs.

Frontiers in Neuroscience, 2020

Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.

Frontiers in Neuroscience, 2020

Spiking Neural Networks (SNNs) may offer an energy-efficient alternative for implementing deep learning applications. In recent years, there have been several proposals focused on supervised (conversion, spike-based gradient descent) and unsupervised (spike timing dependent plasticity) training methods to improve the accuracy of SNNs on large-scale tasks. However, each of these methods suffer from scalability, latency and accuracy limitations. In this paper, we propose novel algorithmic techniques of modifying the SNN configuration with backward residual connections, stochastic softmax and hybrid artificial-and-spiking neuronal activations to improve the learning ability of the training methodologies to yield competitive accuracy, while, yielding large efficiency gains over their artificial counterparts. Note, artificial counterparts refer to conventional deep learning/artificial neural networks. Our techniques apply to VGG/Residual architectures, and are compatible with all forms of training methodologies. Our analysis reveals that the proposed solutions yield near state-of-the-art accuracy with significant energy-efficiency and reduced parameter overhead translating to hardware improvements on complex visual recognition tasks, such as, CIFAR10, Imagenet datatsets.

Neurocomputing, 2021

Spiking Neural Networks (SNNs) are being explored to emulate the astounding capabilities of human brain that can learn and compute functions robustly and efficiently with noisy spiking activities. A variety of spiking neuron models have been proposed to resemble biological neuronal functionalities. With varying levels of bio-fidelity, these models often contain a leak path in their internal states, called membrane potentials. While the leaky models have been argued as more bio-plausible, a comparative analysis between models with and without leak from a purely computational point of view demands attention. In this paper, we investigate the questions regarding the justification of leak and the pros and cons of using leaky behavior. Our experimental results reveal that leaky neuron model provides improved robustness and better generalization compared to models with no leak. However, leak decreases the sparsity of computation contrary to the common notion. Through a frequency domain analysis, we demonstrate the effect of leak in eliminating the high-frequency components from the input, thus enabling SNNs to be more robust against noisy spike-inputs. * equal contribution, authors listed in alphabetical order of surname.

Mobile Networks and Applications, 2019

Current advancement in Internet-of-Things, Cyber-Physical Systems, Cloud-of-Things, and Edge-of-Things technologies have enabled us to design more advanced and event-sensitive real-time monitoring solutions. IoT-assisted healthcare systems need local data processing environment for effective decision making. In the proposed study, a novel edge analytics-assisted monitoring solution is proposed to monitor several physical activities of the patient to determine physical inactivity from their daily routine. Wearable sensors are utilized to monitor physical movements. The main objective of the proposed study is to calculate the scale of the physical inactivity of the patient to make real-time health suggestions. Graphical Processing Unit (GPU) enabled edge nodes are utilized for efficient data processing. An application scenario is proposed to validate the ideology of the proposed system in the healthcare environment. The performance is compared with both machine learning and deep learning-based approaches to justify the proposed system. iFogSim simulator is utilized to simulate the proposed scenario and the performance is validated based on the computation of movement recognition efficiency, network bandwidth efficiency, interoperability, Edge-based data processing reliability and alert generation-based patient security.

2020

SummaryRecent studies hypothesize that motor cortical (MC) dynamics are generated largely through its recurrent connections based on observations that MC activity exhibits rotational structure. However, behavioural and neurophysiological studies suggest that MC behaves like a feedback controller where continuous sensory feedback and interactions with other brain areas contribute substantially to MC processing. We investigated these apparently conflicting theories by building recurrent neural networks that controlled a model arm and received sensory feedback about the limb. Networks were trained to counteract perturbations to the limb and to reach towards spatial targets. Network activities and sensory feedback signals to the network exhibited rotational structure even when the recurrent connections were removed. Furthermore, neural recordings in monkeys performing similar tasks also exhibited rotational structure not only in MC but also in somatosensory cortex. Our results argue tha...

Frontiers in Psychology

Eye-hand coordination of an infant is observed during the early months of their development. Hand regard, which is an example of this coordination, occurs at about 2 months. It is considered that after experiencing hand regard, an infant may recognize their own hands. However, it is unknown how an infant recognizes their hands through hand regard. Accordingly, the process by which an infant recognizes their hands and distinguishes between their hands and other objects was simulated. A simple neural network was trained with a modified real-time recurrent learning (RTRL) algorithm to deal with time-varying input and output during hand regard. The simulation results show that information about recognition of the modeled hands of an infant is stored in cell assemblies, which were self-organized. Cell assemblies appear during the phase of U-shaped developments of hand regard, and the configuration of the cell assemblies changes with each U-shaped development. Furthermore, movements like general movements (GMs) appear during the phase of U-shaped developments of hand regard.

Frontiers in Applied Mathematics and Statistics, 2021

Echo state networks (ESNs) are a powerful form of reservoir computing that only require training of linear output weights while the internal reservoir is formed of fixed randomly connected neurons. With a correctly scaled connectivity matrix, the neurons’ activity exhibits the echo-state property and responds to the input dynamics with certain timescales. Tuning the timescales of the network can be necessary for treating certain tasks, and some environments require multiple timescales for an efficient representation. Here we explore the timescales in hierarchical ESNs, where the reservoir is partitioned into two smaller linked reservoirs with distinct properties. Over three different tasks (NARMA10, a reconstruction task in a volatile environment, and psMNIST), we show that by selecting the hyper-parameters of each partition such that they focus on different timescales, we achieve a significant performance improvement over a single ESN. Through a linear analysis, and under the assum...

Computer Science Review, 2020

The rapid growth of urban populations worldwide imposes new challenges on citizens' daily lives, including environmental pollution, public security, road congestion, etc. New technologies have been developed to manage this rapid growth by developing smarter cities. Integrating the Internet of Things (IoT) in citizens' lives enables the innovation of new intelligent services and applications that serve sectors around the city, including healthcare, surveillance, agriculture, etc. IoT devices and sensors generate large amounts of data that can be analyzed to gain valuable information and insights that help to enhance citizens' quality of life. Deep Learning (DL), a new area of Artificial Intelligence (AI), has recently demonstrated the potential for increasing the efficiency and performance of IoT big data analytics. In this survey, we provide a review of the literature regarding the use of IoT and DL to develop smart cities. We begin by defining the IoT and listing the characteristics of IoT-generated big data. Then, we present the different computing infrastructures used for IoT big data analytics, which include cloud, fog, and edge computing. After that, we survey popular DL models and review the recent research that employs both IoT and DL to develop smart applications and services for smart cities. Finally, we outline the current challenges and issues faced during the development of smart city services.

Water Practice and Technology, 2021

In terms of predicting the flow parameters of a river system, such as discharge and flow depth, the continuity equation plays a vital role. In this research, static- and routing-type dynamic artificial neural networks (ANNs) were incorporated in the multiple sections of a river flow on the basis of a storage parameter. Storage characteristics were presented implicitly and explicitly for various sections in a river system satisfying the continuity norm and mass balance flow. Furthermore, the multiple-input multiple-output (MIMO) model form having two base architectures, namely, MIMO-1 and MIMO-2, was accounted for learning fractional storage and actual storage variations and characteristics in a given model form. The model architecture was also obtained by using a trial-and-error approach, while the network architecture was acquired by employing gamma memory along with use of the multi-layer perceptron model form. Moreover, this paper discusses the comparisons and differences between...

Advances in Artificial Intelligence, 2019

Recurrent Neural Networks (RNN) process sequential data to capture the time-dependency in the input signal. Training a deep RNN conventionally involves segmenting the data sequence to fit the model into memory. Increasing the segment size permits the model to better capture long-term dependencies at the expense of creating larger models that may not fit in memory. Therefore, we introduce a technique to allow designers to train a segmented RNN and obtain the same model parameters as if the entire data sequence was applied regardless of the segment size. This enables an optimal capturing of long-term dependencies. This technique can increase the computational complexity during training. Hence, the proposed technique grants designers the flexibility of balancing memory and runtime requirements. To evaluate the proposed method, we compared the total loss achieved on the testing dataset after every epoch while varying the size of the segments. The results we achieved show matching loss graphs irrespective of the segment size.

IEEE/ACM Transactions on Audio, Speech, and Language Processing

Deep neural networks have been successfully used in the task of black-box modeling of analog audio effects such as distortion. Improving the processing speed and memory requirements of the inference step is desirable to allow such models to be used on a wide range of hardware and concurrently with other software. In this paper, we propose a new application of recent advancements in neural network pruning methods to recurrent black-box models of distortion effects using a Long Short-Term Memory architecture. We compare the efficacy of the method on four different datasets; one distortion pedal and three vacuum tube amplifiers. Iterative magnitude pruning allows us to remove over 99% of parameters from some models without a loss of accuracy. We evaluate the real-time performance of the pruned models and find that a 3x-4x speedup can be achieved, compared to an unpruned baseline. We show that training a larger model and then pruning it outperforms an unpruned model of equivalent hidden size. A listening test confirms that pruning does not degrade the perceived sound quality, but may even slightly improve it. The proposed techniques can be used to design computationally efficient deep neural networks for processing the sound of the electric guitar in real time.

Journal of Intelligent Information Systems, 2018

Social network users publicly share a wide variety of information with their followers and the general public ranging from their opinions, sentiments and personal life activities. There has already been significant advance in analyzing the shared information from both micro (individual user) and macro (community level) perspectives, giving access to actionable insight about user and community behaviors. The identification of personal life events from user's profiles is a challenging yet important task, which if done appropriately, would facilitate more accurate identification of users' preferences, interests and attitudes. For instance, a user who has just broken his phone, is likely to be upset and also be looking to purchase a new phone. While there is work that identifies tweets that include mentions of personal life events, our work in this paper goes beyond the state of the art by predicting a future personal life event that a user will be posting about on Twitter solely based on the past tweets. We propose two architectures based on recurrent neural networks, namely the classification and generation architectures, that determine the future personal life event of a user. We evaluate our work based on a gold standard Twitter life event dataset and compare our work with the state of the art baseline technique for life event detection. While presenting performance measures, we also discuss the limitations of our work in this paper.

Proceedings of the 40th International Conference on Software Engineering, 2018

Automatic correction of programs is a challenging problem with numerous real world applications in security, verification, and education. One application that is becoming increasingly important is the correction of student submissions in online courses for providing feedback. Most existing program repair techniques analyze Abstract Syntax Trees (ASTs) of programs, which are unfortunately unavailable for programs with syntax errors. In this paper, we propose a novel Neuro-symbolic approach that combines neural networks with constraint-based reasoning. Specifically, our method first uses a Recurrent Neural Network (RNN) to perform syntax repairs for the buggy programs; subsequently, the resulting syntactically-fixed programs are repaired using constraint-based techniques to ensure functional correctness. The RNNs are trained using a corpus of syntactically correct submissions for a given programming assignment, and are then queried to fix syntax errors in an incorrect programming submission by replacing or inserting the predicted tokens at the error location. We evaluate our technique on a dataset comprising of over 14,500 student submissions with syntax errors. Our method is able to repair syntax errors in 60% (8689) of submissions, and finds functionally correct repairs for 23.8% (3455) submissions. CCS CONCEPTS • Computing methodologies → Neural networks; • Software and its engineering → Functionality; Search-based software engineering;

Scientific Reports

Room-temperature semiconductor radiation detectors (RTSD) have broad applications in medical imaging, homeland security, astrophysics and others. RTSDs such as CdZnTe, CdTe are often pixelated, and characterization of these detectors at micron level can benefit 3-D event reconstruction at sub-pixel level. Material defects alongwith electron and hole charge transport properties need to be characterized which requires several experimental setups and is labor intensive. The current state-of-art approaches characterize each detector pixel, considering the detector in bulk. In this article, we propose a new microscopic learning-based physical models of RTSD based on limited data compared to what is dictated by the physical equations. Our learning models uses a physical charge transport considering trapping centers. Our models learn these material properties in an indirect manner from the measurable signals at the electrodes and/or free and/or trapped charges distributed in the RTSD for e...

International Journal of Production Research, 2020

A semiconductor distributor that plays a third-party role in the supply chain will buy diverse components from different suppliers, warehouse and resell them to a number of electronics manufacturers with vendor-managed inventories, while suffering both risks of oversupply and shortage due to demand uncertainty. However, demand fluctuation and supply chain complexity are increasing due to shortening product life cycle in the consumer electronics era and long lead time for capacity expansion for high-tech manufacturing. Focusing realistic needs of a leading distributor for semiconductor components and modules, this study aims to construct a UNISON framework based on deep reinforcement learning (RL) for dynamically selecting the optimal demand forecast model for each of the products with the corresponding demand patterns to empower smart production for Industry 3.5. Deep RL that integrates deep learning architecture and RL algorithm can learn successful policies from the dynamic and complex real world. The reward function mechanism of deep RL can reduce negative impact of demand uncertainty. An empirical study was conducted for validation showing practical viability of the proposed approach. Indeed, the developed solution has been in real settings.

Applied Soft Computing, 2021

Sensors

The recent wave of digitalization is characterized by the widespread deployment of sensors in many different environments, e.g., multi-sensor systems represent a critical enabling technology towards full autonomy in industrial scenarios. Sensors usually produce vast amounts of unlabeled data in the form of multivariate time series that may capture normal conditions or anomalies. Multivariate Time Series Anomaly Detection (MTSAD), i.e., the ability to identify normal or irregular operative conditions of a system through the analysis of data from multiple sensors, is crucial in many fields. However, MTSAD is challenging due to the need for simultaneous analysis of temporal (intra-sensor) patterns and spatial (inter-sensor) dependencies. Unfortunately, labeling massive amounts of data is practically impossible in many real-world situations of interest (e.g., the reference ground truth may not be available or the amount of data may exceed labeling capabilities); therefore, robust unsupe...

Neural Computing & Applications, 1997

Conventional adaptive control techniques have for the most part been based on methods for linear or weakly non-linear systems. More recently neural network and genetic algorithm controllers have started to be applied to complex, non-linear dynamic systems. The control of chaotic dynamic systems poses a series of especially challenging problems. In this paper an adaptive control architecture using neural networks and genetic algorithms is applied to a complex, highly nonlinear, chaotic dynamic system: the adaptive attitude control problem (for a satellite), in the presence of large, external forces (which left to themselves lead the system into a chaotic motion). In contrast to the OGY method, which uses small control adjustments to stabilize a chaotic system in an otherwise unstable but natural periodic orbit of the system, the neuro-genetic controller may use large control adjustments and proves capable of effectively attaining any specified system state, with no a priori knowledge of the dynamics, even in the presence of significant noise.

Earth System Science Data

Since the beginning of this century, Europe has been experiencing severe drought events (2003, 2007, 2010, 2018 and 2019) which have had adverse impacts on various sectors, such as agriculture, forestry, water management, health and ecosystems. During the last few decades, projections of the impact of climate change on hydroclimatic extremes have often been used for quantification of changes in the characteristics of these extremes. Recently, the research interest has been extended to include reconstructions of hydroclimatic conditions to provide historical context for present and future extremes. While there are available reconstructions of temperature, precipitation, drought indicators, or the 20th century runoff for Europe, multi-century annual runoff reconstructions are still lacking. In this study, we have used reconstructed precipitation and temperature data, Palmer Drought Severity Index and available observed runoff across 14 European catchments in order to develop annual runoff reconstructions for the period 1500-2000 using two data-driven and one conceptual lumped hydrological model. The comparison to observed runoff data has shown a good match between the reconstructed and observed runoff and their characteristics, particularly deficit volumes. On the other hand, the validation of input precipitation fields revealed an underestimation of the variance across most of Europe, which is propagated into the reconstructed runoff series. The reconstructed runoff is available via Figshare, an open-source scientific data repository, under the DOI

Chaos: An Interdisciplinary Journal of Nonlinear Science, 2020

The Deep Time-Delay Reservoir Computing concept utilizes unidirectionally connected systems with timedelays for supervised learning. We present how the dynamical properties of a deep Ikeda-based reservoir are related to its memory capacity (MC) and how that can be used for optimization. In particular, we analyze bifurcations of the corresponding autonomous system and compute conditional Lyapunov exponents, which measure the generalized synchronization between the input and the layer dynamics. We show how the MC is related to the systems distance to bifurcations or magnitude of the conditional Lyapunov exponent. The interplay of different dynamical regimes leads to an adjustable distribution between linear and nonlinear MC. Furthermore, numerical simulations show resonances between clock cycle and delays of the layers in all degrees of the MC. Contrary to MC losses in single-layer reservoirs, these resonances can boost separate degrees of the MC and can be used, e.g. to design a system with maximum linear MC. Accordingly, we present two configurations that empower either high nonlinear MC or long time linear MC.

Neural Computing and Applications, 2015

A multilayered neural network is a multi-input, multi-output nonlinear system in which network weights can be trained by using parameter estimation algorithms. In this paper, a novel training method is proposed. This method is based on the relatively new smooth variable structure filter (SVSF) and is formulated for feed-forward multilayer perceptron training. The SVSF is a state and parameter estimation that is based on the sliding mode concept and works in a predictor-corrector fashion. The SVSF training performance is tested on three benchmark pattern classification problems. Furthermore, a study is presented comparing the popular back-propagation method, the extended Kalman filter, and the SVSF.

Scientific Reports

Given the inherent complexity of the human nervous system, insight into the dynamics of brain activity can be gained from studying smaller and simpler organisms. While some of the potential target organisms are simple enough that their behavioural and structural biology might be well-known and understood, others might still lead to computationally intractable models that require extensive resources to simulate. Since such organisms are frequently only acting as proxies to further our understanding of underlying phenomena or functionality, often one is not interested in the detailed evolution of every single neuron in the system. Instead, it is sufficient to observe the subset of neurons that capture the effect that the profound nonlinearities of the neuronal system have in response to different stimuli. In this paper, we consider the well-known nematode Caenorhabditis elegans and seek to investigate the possibility of generating lower complexity models that capture the system’s dyna...

Energies

Smart meters (SMs) can play a key role in monitoring vital aspects of different applications such as smart grids (SG), alternative currents (AC) optimal power flows, adversarial training, time series data, etc. Several practical privacy implementations of SM have been made in the literature, but more studies and testing may be able to further improve efficiency and lower implementation costs. The major objectives of cyberattacks are the loss of data privacy on SM-based SG/power grid (PG) networks and threatening human life. As a result, losing data privacy is very expensive and gradually hurts the national economy. Consequently, employing an efficient trust model against cyberattacks is strictly desired. This paper presents a research pivot for researchers who are interested in security and privacy and shade light on the importance of the SM. We highlight the involved SMs’ features in several applications. Afterward, we focus on the SMs’ vulnerabilities. Then, we consider eleven tru...

Proceedings of the ACM on Measurement and Analysis of Computing Systems

4G, 5G, and smart city networks often rely on microwave and millimeter-wave x-haul links. A major challenge associated with these high frequency links is their susceptibility to weather conditions. In particular, precipitation may cause severe signal attenuation, which significantly degrades the network performance. In this paper, we develop a Predictive Network Reconfiguration (PNR) framework that uses historical data to predict the future condition of each link and then prepares the network ahead of time for imminent disturbances. The PNR framework has two components: (i) an Attenuation Prediction (AP) mechanism; and (ii) a Multi-Step Network Reconfiguration (MSNR) algorithm. The AP mechanism employs an encoder-decoder Long Short-Term Memory (LSTM) model to predict the sequence of future attenuation levels of each link. The MSNR algorithm leverages these predictions to dynamically optimize routing and admission control decisions aiming to maximize network utilization, while preser...

Computational Economics, 2020

Monitoring liquidity management of banks is one of the prime tasks of central banks. A bank that manages its liquidity inadequately can severely harm its liquidity position and potentially threaten the stability of the entire financial system. Central banks try to anticipate these risks by carefully monitoring the liquidity management of banks in large-value payment systems (LVPSs). Typically, they do this based on statistical methods in which various risk indicators related to the liquidity usage of banks are calculated from the transaction log of an LVPS. These indicators need to be manually analyzed by payment experts to find irregularities that could signal potential risks. Although statistical methods provide much insight into the liquidity management of banks, they do not scale well to the large number of banks that are subject to risk monitoring and the high velocity by which payments are nowadays settled. In this paper, we investigate whether the liquidity management of banks can be monitored more efficiently by anomaly detection. We construct different probabilistic classifiers that classify delta sequences of banks by the corresponding bank. A delta sequence captures the change in the liquidity position of a bank in an LVPS throughout a given day. Accordingly, anomalies in the intraday liquidity usage of banks are detected by determining whether the classifiers misclassify recent delta sequences that were not used to train the classifiers. Our results show that recurrent neural networks are well suited to perform this classification task and detect many irregularities in payment behavior that are interesting for the supervisors and operators of an LVPS.

Bulletin of Earthquake Engineering

Within seismology, geology, civil and structural engineering, deep learning (DL), especially via generative adversarial networks (GANs), represents an innovative, engaging, and advantageous way to generate reliable synthetic data that represent actual samples’ characteristics, providing a handy data augmentation tool. Indeed, in many practical applications, obtaining a significant number of high-quality information is demanding. Data augmentation is generally based on artificial intelligence (AI) and machine learning data-driven models. The DL GAN-based data augmentation approach for generating synthetic seismic signals revolutionized the current data augmentation paradigm. This study delivers a critical state-of-art review, explaining recent research into AI-based GAN synthetic generation of ground motion signals or seismic events, and also with a comprehensive insight into seismic-related geophysical studies. This study may be relevant, especially for the earth and planetary scien...

Energies

With the steady increase in the use of renewable energy sources in the energy sector, new challenges arise, especially the unpredictability of these energy sources. This uncertainty complicates the management, planning, and development of energy systems. An effective solution to these challenges is short-term forecasting of the output of photovoltaic power plants. In this paper, a novel method for short-term production prediction was explored which involves continuous photography of the sky above the photovoltaic power plant. By analyzing a series of sky images, patterns can be identified to help predict future photovoltaic power generation. A hybrid model that integrates both a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) for short-term production forecasting was developed and tested. This model effectively detects spatial and temporal patterns from images and power output data, displaying considerable prediction accuracy. In particular, a 74% correlation ...

2021

Background: Time-lapse microscopy has been widely used in biomedical experiments because it can visualize the molecular activities of living cells in real time. However, biomedical researchers are still conducting cell lineage analysis manually. Developing automatic lineage tracing algorithms is a challenging task. In the past two decades, deep neural networks (DNNs) became have shown outstanding performance on computer vision tasks. They can learn complex visual features, capture long-range temporal dependencies, and have the potential to be used for automatic cell lineage analysis. Methods: In this study, we propose a multi-task spatio-temporal feature based deep neural network for cell lineages analysis (Cell-STN). The Cell-STN extracts spatio-temporal features from microscopy image sequences by leveraging our convolutional long short-term memory based core block. And the proposed Cell-STN utilized a task specific network to predict the cell location, the mitosis event, and the a...

International Journal of Software Engineering and Knowledge Engineering, 2009

Predicting changes in software entities (e.g. source files) that are more likely to change can help in the efficient allocation of the project resources. A powerful change prediction tool can improve maintenance and evolution tasks in software projects in terms of cost and time factors. The vast majority of research works have focused on determining "where" the most change-prone entities are, and "how" the change will be propagated through a system. This article suggests that knowing "when" changes are likely to happen can also provide another consideration for managers and developers to plan their maintenance activities more efficiently. To address this issue, a Neural Network-based Temporal Change Prediction (NNTCP) framework is proposed. This novel framework indicates "where" the changes are likely to happen (i.e. hot spots), and then adds the time dimension to predict "when" it may occur. In proving this concept, the NNTCP framew...

International Journal of Information System Modeling and Design, 2017

This article describes how sequential data modeling is a relevant task in Cybersecurity. Sequences are attributed temporal characteristics either explicitly or implicitly. Recurrent neural networks (RNNs) are a subset of artificial neural networks (ANNs) which have appeared as a powerful, principle approach to learn dynamic temporal behaviors in an arbitrary length of large-scale sequence data. Furthermore, stacked recurrent neural networks (S-RNNs) have the potential to learn complex temporal behaviors quickly, including sparse representations. To leverage this, the authors model network traffic as a time series, particularly transmission control protocol / internet protocol (TCP/IP) packets in a predefined time range with a supervised learning method, using millions of known good and bad network connections. To find out the best architecture, the authors complete a comprehensive review of various RNN architectures with its network parameters and network structures. Ideally, as a t...

Biomimetics

One developing approach for robotic control is the use of networks of dynamic neurons connected with conductance-based synapses, also known as Synthetic Nervous Systems (SNS). These networks are often developed using cyclic topologies and heterogeneous mixtures of spiking and non-spiking neurons, which is a difficult proposition for existing neural simulation software. Most solutions apply to either one of two extremes, the detailed multi-compartment neural models in small networks, and the large-scale networks of greatly simplified neural models. In this work, we present our open-source Python package SNS-Toolbox, which is capable of simulating hundreds to thousands of spiking and non-spiking neurons in real-time or faster on consumer-grade computer hardware. We describe the neural and synaptic models supported by SNS-Toolbox, and provide performance on multiple software and hardware backends, including GPUs and embedded computing platforms. We also showcase two examples using the ...

Advances in intelligent systems and computing, 2021

PLOS ONE, 2022

The COVID-19 pandemic continues to have major impact to health and medical infrastructure, economy, and agriculture. Prominent computational and mathematical models have been unreliable due to the complexity of the spread of infections. Moreover, lack of data collection and reporting makes modelling attempts difficult and unreliable. Hence, we need to re-look at the situation with reliable data sources and innovative forecasting models. Deep learning models such as recurrent neural networks are well suited for modelling spatiotemporal sequences. In this paper, we apply recurrent neural networks such as long short term memory (LSTM), bidirectional LSTM, and encoder-decoder LSTM models for multi-step (short-term) COVID-19 infection forecasting. We select Indian states with COVID-19 hotpots and capture the first (2020) and second (2021) wave of infections and provide two months ahead forecast. Our model predicts that the likelihood of another wave of infections in October and November ...

Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018

Semi-supervised learning is a branch of machine learning techniques that aims to make fully use of both labeled and unlabeled instances to improve the prediction performance. The size of modern real world datasets is ever-growing so that acquiring label information for them is extraordinarily difficult and costly. Therefore, deep semi-supervised learning is becoming more and more popular. Most of the existing deep semi-supervised learning methods are built under the generative model based scheme, where the data distribution is approximated via input data reconstruction. However, this scheme does not naturally work on discrete data, e.g., text; in addition, learning a good data representation is sometimes directly opposed to the goal of learning a high performance prediction model. To address the issues of this type of methods, we reformulate the semi-supervised learning as a model-based reinforcement learning problem and propose an adversarial networks based framework. The proposed framework contains two networks: a predictor network for target estimation and a judge network for evaluation. The judge network iteratively generates proper reward to guide the training of predictor network, and the predictor network is trained via policy gradient. Based on the aforementioned framework, we propose a recurrent neural network based model for semi-supervised text classification. We conduct comprehensive experimental analysis on several real world benchmark text datasets, and the results from our evaluations show that our method outperforms other competing state-of-the-art methods.

Bio-inspired Modeling of Cognitive Tasks, 2007

Motivation: Predicting HIV resistance to drugs is one of many problems for which bioinformaticians have implemented and trained machine learning methods, such as neural networks. Predicting HIV resistance would be much easier if we could directly use the three-dimensional (3D) structure of the targeted protein sequences, but unfortunately we rarely have enough structural information available to train a neural network. Furthermore, prediction of the 3D structure of a protein is not straightforward. However, characteristics related to the 3D structure can be used to train a machine learning algorithm as an alternative to take into account the information of the protein folding in the 3D space. Here, starting from this philosophy, we select the amino acid energies as features to predict HIV drug resistance, using a specific topology of a neural network. Results: In this paper, we demonstrate that the amino acid energies are good features to represent the HIV genotype. In addition, it was shown that Bidirectional Recurrent Neural Networks can be used as an efficient classification method for this problem. The prediction performance that was obtained was greater than or at least comparable to results obtained previously. The accuracies vary between 81.3% and 94.7%.

Computational Intelligence and Neuroscience, 2015

Artificial Neural Network (ANN) design is a complex task because its performance depends on the architecture, the selected transfer function, and the learning algorithm used to train the set of synaptic weights. In this paper we present a methodology that automatically designs an ANN using particle swarm optimization algorithms such as Basic Particle Swarm Optimization (PSO), Second Generation of Particle Swarm Optimization (SGPSO), and a New Model of PSO called NMPSO. The aim of these algorithms is to evolve, at the same time, the three principal components of an ANN: the set of synaptic weights, the connections or architecture, and the transfer functions for each neuron. Eight different fitness functions were proposed to evaluate the fitness of each solution and find the best design. These functions are based on the mean square error (MSE) and the classification error (CER) and implement a strategy to avoid overtraining and to reduce the number of connections in the ANN. In additi...

Applied sciences, 2022

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY

Array, 2021

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Machine Learning, 2010

In the last decade, connectionist models have been proposed that can process structured information directly. These methods, which are based on the use of graphs for the representation of the data and the relationships within the data, are particularly suitable for handling relational learning tasks. In this paper, two recently proposed architectures of this kind, i.e. Graph Neural Networks (GNNs) and Relational Neural Networks (RelNNs), are compared and discussed, along with their corresponding learning schemes. The goal is to evaluate the performance of these methods on benchmarks that are commonly used by the relational learning community. Moreover, we also aim at reporting differences in the behavior of the two models, in order to gain insights on possible extensions of the approaches. Since RelNNs have been developed with the specific task of learning aggregate functions in mind, some experiments are run considering that particular task. In addition, we carry out more general experiments on the mutagenesis and the biodegradability datasets, on which several other relational learners have been evaluated. The experimental results are promising and suggest that RelNNs and GNNs can be a viable approach for learning on relational data.

Journal of Informetrics

Prediction of the future performance of academic journals is a task that can benefit a variety of stakeholders including editorial staff, publishers, indexing services, researchers, university administrators and granting agencies. Using historical data on journal performance, this can be framed as a machine learning regression problem. In this work, we study two such regression tasks: 1) prediction of the number of citations a journal will receive during the next calendar year, and 2) prediction of the Elsevier CiteScore a journal will be assigned for the next calendar year. To address these tasks, we first create a dataset of historical bibliometric data for journals indexed in Scopus. We propose the use of neural network models trained on our dataset to predict the future performance of journals. To this end, we perform feature selection and model configuration for a Multi-Layer Perceptron and a Long Short-Term Memory. Through experimental comparisons to heuristic prediction baselines and classical machine learning models, we demonstrate superior performance in our proposed models for the prediction of future citation and CiteScore values.

Scientific Reports, 2021

A central question in neuroscience is how self-organizing dynamic interactions in the brain emerge on their relatively static structural backbone. Due to the complexity of spatial and temporal dependencies between different brain areas, fully comprehending the interplay between structure and function is still challenging and an area of intense research. In this paper we present a graph neural network (GNN) framework, to describe functional interactions based on the structural anatomical layout. A GNN allows us to process graph-structured spatio-temporal signals, providing a possibility to combine structural information derived from diffusion tensor imaging (DTI) with temporal neural activity profiles, like that observed in functional magnetic resonance imaging (fMRI). Moreover, dynamic interactions between different brain regions discovered by this data-driven approach can provide a multi-modal measure of causal connectivity strength. We assess the proposed model's accuracy by evaluating its capabilities to replicate empirically observed neural activation profiles, and compare the performance to those of a vector auto regression (VAR), like that typically used in Granger causality. We show that GNNs are able to capture long-term dependencies in data and also computationally scale up to the analysis of large-scale networks. Finally we confirm that features learned by a GNN can generalize across MRI scanner types and acquisition protocols, by demonstrating that the performance on small datasets can be improved by pre-training the GNN on data from an earlier study. We conclude that the proposed multi-modal GNN framework can provide a novel perspective on the structure-function relationship in the brain. Accordingly this approach appears to be promising for the characterization of the information flow in brain networks.

Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016

Precise item categorization is a key issue in e-commerce domains. However, it still remains a challenging problem due to data size, category skewness, and noisy metadata. Here, we demonstrate a successful report on a deep learning-based item categorization method, i.e., deep categorization network (DeepCN), in an ecommerce website. DeepCN is an end-to-end model using multiple recurrent neural networks (RNNs) dedicated to metadata attributes for generating features from text metadata and fully connected layers for classifying item categories from the generated features. The categorization errors are propagated back through the fully connected layers to the RNNs for weight update in the learning process. This deep learning-based approach allows diverse attributes to be integrated into a common representation, thus overcoming sparsity and scalability problems. We evaluate DeepCN on large-scale real-world data including more than 94 million items with approximately 4,100 leaf categories from a Korean e-commerce website. Experiment results show our method improves the categorization accuracy compared to the model using single RNN as well as a standard classification model using unigram-based bag-of-words. Furthermore, we investigate how much the model parameters and the used attributes influence categorization performances. CCS Concepts • Computing methodologies ➝ Neural networks • Computing methodologies ➝ Supervised learning by classification • Information systems ➝ Online shopping.

Microwave and Optical Technology Letters, 2001

Figure 5 Measured antenna gains for the antenna studied in Fig-Ž. Ž. ure 2 with ␣ s 12Њ. a f s 1875 MHz. b f s 2100 MHz 1 2 by inserting two narrow slits at one of the radiating edges of the patch. The two operating frequencies have the same polarization planes, and the frequency ratio of the two frequencies is tunable from 1.08 to 1.17 by varying the flare angle of the trapezoid microstrip-line section. In addition, the optimal feed positions are found to be insensitive to the variation in the flare angle, which makes the present design easy to implement.

PLOS Computational Biology, 2021

Sequential behaviour is often compositional and organised across multiple time scales: a set of individual elements developing on short time scales (motifs) are combined to form longer functional sequences (syntax). Such organisation leads to a natural hierarchy that can be used advantageously for learning, since the motifs and the syntax can be acquired independently. Despite mounting experimental evidence for hierarchical structures in neuroscience, models for temporal learning based on neuronal networks have mostly focused on serial methods. Here, we introduce a network model of spiking neurons with a hierarchical organisation aimed at sequence learning on multiple time scales. Using biophysically motivated neuron dynamics and local plasticity rules, the model can learn motifs and syntax independently. Furthermore, the model can relearn sequences efficiently and store multiple sequences. Compared to serial learning, the hierarchical model displays faster learning, more flexible relearning, increased capacity, and higher robustness to perturbations. The hierarchical model redistributes the variability: it achieves high motif fidelity at the cost of higher variability in the between-motif timings.

PLOS Computational Biology, 2020

Learning to produce spatiotemporal sequences is a common task that the brain has to solve. The same neurons may be used to produce different sequential behaviours. The way the brain learns and encodes such tasks remains unknown as current computational models do not typically use realistic biologically-plausible learning. Here, we propose a model where a spiking recurrent network of excitatory and inhibitory spiking neurons drives a read-out layer: the dynamics of the driver recurrent network is trained to encode time which is then mapped through the read-out neurons to encode another dimension, such as space or a phase. Different spatiotemporal patterns can be learned and encoded through the synaptic weights to the read-out neurons that follow common Hebbian learning rules. We demonstrate that the model is able to learn spatiotemporal dynamics on time scales that are behaviourally relevant and we show that the learned sequences are robustly replayed during a regime of spontaneous activity.

2023

Spiking Neural Networks (SNNs) have attracted recent interest due to their energy efficiency and biological plausibility. However, the performance of SNNs still lags behind traditional Artificial Neural Networks (ANNs), as there is no consensus on the best learning algorithm for SNNs. Best-performing SNNs are based on ANN to SNN conversion or learning with spike-based backpropagation through surrogate gradients. The focus of recent research has been on developing and testing different learning strategies, with hand-tailored architectures and parameter tuning. Neuroevolution (NE), has proven successful as a way to automatically design ANNs and tune parameters, but its applications to SNNs are still at an early stage. DENSER is a NE framework for the automatic design and parametrization of ANNs, based on the principles of Genetic Algorithms (GA) and Structured Grammatical Evolution (SGE). In this paper, we propose SPENSER, a NE framework for SNN generation based on DENSER, for image classification on the MNIST and Fashion-MNIST datasets. SPENSER generates competitive performing networks with a test accuracy of 99.42% and 91.65% respectively. • Computing methodologies → Neural networks; Computer vision; • Theory of computation → Evolutionary algorithms; Grammars and context-free languages.

Desalination and Water Treatment, 2018

In this study, the biosorption performance of banana floret was assessed as a new biosorbent for the removal of Cu(II) ions (a model heavy metal) from aqueous solutions. Batch experiments were conducted to assess the effects of agitation, particle size, pH, temperature and initial concentration. Kinetic and equilibrium data were modeled, and mass transfer studies were conducted to elucidate the mechanisms of biosorption. Kinetic data were best simulated using the diffusion-chemisorption model while equilibrium data were best represented by the Sips isotherm. The dominant transport mechanism was attributed to intraparticle diffusion while the dominant attachment mechanism was chemical sorption. A predictive model was successfully developed using an artificial neural network (ANN) and optimized using a genetic algorithm (GA). The accuracy of the ANN-GA prediction was validated by laboratory experiments, which revealed a residual error of 1.3% and thus underscores the applicability of the model. This new biosorbent exhibited a remarkable affinity for the heavy metal ion and compared well to other reported biosorbents in the literature.

Neuroinformatics, 2013

The mirror neuron system in primates matches observations of actions with the motor representations used for their execution, and is a topic of intense research and debate in biological and computational disciplines. In robotics, models of this system have been used for enabling robots to imitate and learn how to perform tasks from human demonstrations. Yet, existing computational and robotic models of these systems are found in multiple levels of description, and although some models offer plausible explanations and testable predictions, the difference in the granularity of the experimental setups, methodologies, computational structures and selected modeled data make principled meta-analyses, common in other fields, difficult. In this paper, we adopt an interdisciplinary approach, using the BODB integrated environment in order to bring together several different but complementary computational models, by functionally decomposing them into brain operating principles (BOPs) which each capture a limited subset of the model's functionality. We then explore links from these BOPs to neuroimaging and neurophysiological data in order to pinpoint complementary and conflicting explanations and compare predictions against selected sets of neurobiological data. The results of this comparison are used to interpret mirror system neuroimaging results in terms of neural network activity, evaluate the biological plausibility of mirror system models, and suggest new experiments that can shed light on the neural basis of mirror systems.

European Radiology, 2020

Objectives A convolutional neural network (CNN) was adapted to automatically detect early-stage nasopharyngeal carcinoma (NPC) and discriminate it from benign hyperplasia on a non-contrast-enhanced MRI sequence for potential use in NPC screening programs. Methods We retrospectively analyzed 412 patients who underwent T2-weighted MRI, 203 of whom had biopsy-proven primary NPC confined to the nasopharynx (stage T1) and 209 had benign hyperplasia without NPC. Thirteen patients were sampled randomly to monitor the training process. We applied the Residual Attention Network architecture, adapted for three-dimensional MR images, and incorporated a slice-attention mechanism, to produce a CNN score of 0-1 for NPC probability. Threefold crossvalidation was performed in 399 patients. CNN scores between the NPC and benign hyperplasia groups were compared using Student's t test. Receiver operating characteristic with the area under the curve (AUC) was performed to identify the optimal CNN score threshold. Results In each fold, significant differences were observed in the CNN scores between the NPC and benign hyperplasia groups (p < .01). The AUCs ranged from 0.95 to 0.97 with no significant differences between the folds (p = .35 to .92). The combined AUC from all three folds (n = 399) was 0.96, with an optimal CNN score threshold of > 0.71, producing a sensitivity, specificity, and accuracy of 92.4%, 90.6%, and 91.5%, respectively, for NPC detection. Conclusion Our CNN method applied to T2-weighted MRI could discriminate between malignant and benign tissues in the nasopharynx, suggesting that it as a promising approach for the automated detection of early-stage NPC. • The convolutional neural network (CNN)-based algorithm could automatically discriminate between malignant and benign diseases using T2-weighted fat-suppressed MR images. • The CNN-based algorithm had an accuracy of 91.5% with an area under the receiver operator characteristic curve of 0.96 for discriminating early-stage T1 nasopharyngeal carcinoma from benign hyperplasia. • The CNN-based algorithm had a sensitivity of 92.4% and specificity of 90.6% for detecting early-stage nasopharyngeal carcinoma.