Learning Opposites with Evolving Rules

ArXiv, 2021

We introduce feature alignment, a technique for obtaining approximate reversibility in artificial neural networks. By means of feature extraction, we can train a neural network to learn an estimated map for its reverse process from outputs to inputs. Combined with variational autoencoders, we can generate new samples from the same statistics as the training data. Improvements of the results are obtained by using concepts from generative adversarial networks. Finally, we show that the technique can be modified for training neural networks locally, saving computational memory resources. Applying these techniques, we report results for three vision generative tasks: MNIST, CIFAR-10, and celebA.

2006

A model for learning in the limit is defined where a (so-called iterative) learner gets all positive examples from the target language, tests every new conjecture with a teacher (oracle) if it is a subset of the target language (and if it is not, then it receives a negative counterexample), and uses only limited long-term memory (incorporated in conjectures). Three variants of this model are compared: when a learner receives least negative counterexamples, the ones whose size is bounded by the maximum size of input seen so far, and arbitrary ones. We also compare our learnability model with other relevant models of learnability in the limit, study how our model works for indexed classes of recursive languages, and show that learners in our model can work in non-U-shaped way — never abandoning the first right conjecture.

J. Mach. Learn. Res., 2016

Traditionally the machine-learning community has viewed the No Free Lunch (NFL) theorems for search and optimization as a limitation. We review, analyze, and unify the NFL theorem with the perspectives of "blind" search and meta-learning to arrive at necessary conditions for improving black-box optimization. We survey meta-learning literature to determine when and how meta-learning can benefit machine learning. Then, we generalize meta-learning in the context of the NFL theorems, to arrive at a novel technique called anti-training with sacrificial data (ATSD). Our technique applies at the meta level to arrive at domain specific algorithms. We also show how to generate sacrificial data. An extensive case study is presented along with simulated annealing results to demonstrate the efficacy of the ATSD method.

Knowledge and Information Systems, 2020

This paper is concerned by solving supervised machine learning problem as an inverse problem. Recently, many works have focused on defining a relationship between supervised learning and the well-known inverse problems. However, this connection between the learning problem and the inverse one has been done in the particular case where the inverse problem is reformulated as a minimization problem with a quadratic cost functional (L 2 cost functional). Although, it is well known that the cost functional can be L 1 , L 2 or any positive function that measures the gap between the predicted data and the observed one. Indeed, the use of L 1 loss function for supervised learning problem gives more consistent results (see Rosasco et al. in Neural Comput 16:1063-1076, 2004). This strengthens the idea of reformulating the inverse problem, associated to machine learning problem, into a minimization problem using L 1 functional. However, the L 1 loss function is non-differentiable, which precludes the use of standard optimization tools. To overcome this difficulty, we propose in this paper a new technique of approximation based on the reformulation of the associated inverse problem into a minimizing one of a slanting cost functional Chen et al. (MIS Q Manag Inf Syst 36:1165-1188, 2012), which is solved using Tikhonov regularization and Newton's method. This approach leads to an efficient numerical algorithm allowing us to solve supervised learning problem in the most general framework. To confirm this, we present some numerical results showing the efficiency of the proposed approach. Furthermore, the numerical experiment validation is made through academic and real-life data. Thus, the comparison with existing methods and numerical stability of the algorithm is presented in order to show that our approach is better in terms of convergence speed and quality of predicted models.

lia.ufc.br

The ability to automatically create negative examples is a characteristic that most ILP learning systems do not have. In cases when negative examples are scarce or even not available, it is up to the system to generate them lest the learning may not be carried out. We propose a method to remedy this situation that requires no interaction with the user. The method, based solely on the user-provided positive examples, was implemented in the system Shrinp with promising results. We also show experiments with two well-known systems, Golem and Progol, in which only computer-generated negative examples are used, with significant outcome. Our method is broader than those used by many ILP systems. ABSTRACT The ability to automatically create negative examples is a characteristic that most ILP learning systems do not have. In cases when negative examples are scarce or even not available, it is up to the system to generate them lest the learning may not be carried out. We propose a method to remedy this situation that requires no interaction with the user. The method, based solely on the user-provided positive examples, was implemented in the system Shrinp with promising results. We also show experiments with two well-known systems, Golem and Progol, in which only computer-generated negative examples are used, with significant outcome. Our method is broader than those used by many ILP systems.

IEEE transactions on cybernetics, 2014

Evolutionary algorithms (EAs) are widely employed for solving optimization problems with rugged fitness landscapes. Opposition-based learning (OBL) is a recent tool developed to improve the convergence rate of EAs. In this paper, we derive the probabilities that distances between OBL points and the optimization problem solution are less than the distance between a given EA individual and the optimal solution. We find that the quasi-reflected opposition point yields the highest probability and is the most likely candidate to be closer to the optimal solution. We then employ CEC 2013 competition benchmark problems and select a set of trajectory optimization problems from the European Space Agency to study the performance of three OBL algorithms in conjunction with three different EAs. The CEC 2013 test suit simulations indicate that quasi-reflection accelerates the performance of the EA, especially for more difficult composition functions. The space trajectory experiments reveal that ...

2007

We present a rapid training algorithm for two-layer, feed-forward neural networks. The rst layer of weights is determined by a genetic algorithm, the second layer by an iterated pseudo-inverse. We apply our method to problems such as parity and encoder-decoder (the latter is of the form \N{M{N," with M < N, in which the input and output of the net are identical). The network's form is: y = 1 Vx followed by z = 2 Wy, where x, y, and z are the input, hidden, and output vectors; V and W are the weights matrices; 1 is a piecewise linear function, and 2 is sign. A genetic algorithm evolves V using a tness function based on the network's performance after W has been determined by an iterated pseudoinverse.

IEEE Transactions on Evolutionary Computation, 2000

Evolutionary algorithms (EAs) are well-known optimization approaches to deal with nonlinear and complex problems. However, these population-based algorithms are computationally expensive due to the slow nature of the evolutionary process. This paper presents a novel algorithm to accelerate the differential evolution (DE). The proposed opposition-based DE (ODE) employs opposition-based learning (OBL) for population initialization and also for generation jumping. In this work, opposite numbers have been utilized to improve the convergence rate of DE. A comprehensive set of 58 complex benchmark functions including a wide range of dimensions is employed for experimental verification. The influence of dimensionality, population size, jumping rate, and various mutation strategies are also investigated. Additionally, the contribution of opposite numbers is empirically verified. We also provide a comparison of ODE to fuzzy adaptive DE (FADE). Experimental results confirm that the ODE outperforms the original DE and FADE in terms of convergence speed and solution accuracy.

2006

Learning and the ability to learn are important factors in development and evolutionary processes [1]. Depending on the level, the complexity of learning can strongly vary. While associative learning can explain simple learning behaviour [1,2] much more sophisticated strategies seem to be involved in complex learning tasks. This is particularly evident in machine learning theory [3] (reinforcement learning [4], statistical learning [5]), but it equally shows up in trying to model natural learning behaviour [2]. A general setting for modelling learning processes in which statistical aspects are relevant is provided by the neural network (NN) paradigm. This is in particular of interest for natural, learning by experience situations. NN learning models can incorporate elementary learning mechanisms based on neuro-physiological analogies, such as the Hebb rule, and lead to quantitative results concerning the dynamics of the learning process [6]. The Hebb rule, however, cannot be directl...

Brazilian Journal of Physics, 1998

We investigate the learning of a rule from examples of the case of boolean perceptron. Previous studies of this problem have been made using the full quenched theory. We consider here two alternative approaches that can be applied easily. The two-replicas interactions approach considerably improves upon the well-known rst-order approach. The mean eld approach proved some results that have been obtained previously using the complex full quenched theory. Both approximations have been applied to both continuous weights and discrete weights perceptron.