Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Related Work
Method
Datasets
Results
Discussion
References
All Topics
Computer Science

How Important is Weight Symmetry in Backpropagation?

Profile image of TOMASO POGGIOTOMASO POGGIO

2015, arXiv (Cornell University)

https://doi.org/10.48550/ARXIV.1510.05067
visibility

…

description

9 pages

link

1 file

Abstract

Gradient backpropagation (BP) requires symmetric feedforward and feedback connections-the same weights must be used for forward and backward passes. This "weight transport problem" (Grossberg 1987) is thought to be one of the main reasons to doubt BP's biologically plausibility. Using 15 different classification datasets, we systematically investigate to what extent BP really depends on weight symmetry. In a study that turned out to be surprisingly similar in spirit to Lillicrap et al.'s demonstration (Lillicrap et al. 2014) but orthogonal in its results, our experiments indicate that: (1) the magnitudes of feedback weights do not matter to performance (2) the signs of feedback weights do matter-the more concordant signs between feedforward and their corresponding feedback connections, the better (3) with feedback weights having random magnitudes and 100% concordant signs, we were able to achieve the same or even better performance than SGD. (4) some normalizations/stabilizations are indispensable for such asymmetric BP to work, namely Batch Normalization (BN) (Ioffe and Szegedy 2015) and/or a "Batch Manhattan" (BM) update rule.

Related papers

Revisiting the problem of weight initialization for multi-layer perceptrons trained with back propagation
Dimitrios A Karras, Michael Vrahatis

Advances in Neuro-Information Processing, 2009

View PDFchevron_right
On the benefits of non-linear weight updates
Paul Norridge

Recent work has suggested that the generalisation performance of a DNN is related to the extent to which the Signal-to-Noise Ratio is optimised at each of the nodes. In contrast, Gradient Descent methods do not always lead to SNR-optimal weight configurations. One way to improve SNR performance is to suppress large weight updates and amplify small weight updates. Such balancing is already implicit in some common optimizers, but we propose an approach that makes this explicit. The method applies a non-linear function to gradients prior to making DNN parameter updates. We investigate the performance with such non-linear approaches. The result is an adaptation to existing optimizers that improves performance for many problem types.

View PDFchevron_right
Constraining the topology of neural networks to ensure dynamics with symmetry properties
Rafael Lopes

Physical Review E, 2004

This paper addresses the training of network models from data produced by systems with symmetry properties. It is argued that although general networks are global approximators, in practice some properties such as symmetry are very hard to learn from data. In order to guarantee that the final network will be symmetrical, constraints are developed for two types of models, namely, the multilayer perceptron ͑MLP͒ network and the radial basis function ͑RBF͒ network. In global modeling problems it becomes crucial to impose conditions for symmetry in order to stand a chance of reproducing symmetry-related phenomena. Sufficient conditions are given for MLP and RBF networks to have a set of fixed points that are symmetrical with respect to the origin of the phase space. In the case of MLP networks such conditions reduce to the absence of bias parameters and the requirement of odd activation functions. This turns out to be important from a dynamical point of view since some phenomena are only observed in the context of symmetry, which is not a structurally stable property. The results are illustrated using bench systems that display symmetry, such as the Duffing-Ueda oscillator and the Lorenz system.

View PDFchevron_right
Neural Nonlinear Classifiers with Synaptic Weight Commitment
Konstantinos Diamantaras

AbstTact-We develop a theory for immediately detecting patterns violating linear separability in a training set, as soon as they are presented to the classifier. The theory is based on the computation of the solution cone in the weight space of a single linear threshold unit (LTU). The separability-violating patterns can be skipped and in the end, we obtain a linearly separable subset of the original training set together with the subset's solution cone. We further propose an iterative algorithm for computing the solution cone and we introduce the concept of garbage collection for removing redundancy from the cone representation. The algorithm can be implemented by a novel neural network architecture where the synaptic weights have constant values throughout the life of the synapse. We show that by combining multiple such models we can construct a larger network capable of learning the classification boundaries of convex classes, therefore solving an important class of nonlinearly separable problems.

View PDFchevron_right
WeightScale: Interpreting Weight Change in Neural Networks
ayush agrawal

ArXiv, 2021

Interpreting the learning dynamics of neural networks can provide useful insights into how networks learn and the development of better training and design approaches. We present an approach to interpret learning in neural networks by measuring relative weight change on a per layer basis and dynamically aggregating emerging trends through combination of dimensionality reduction and clustering which allows us to scale to very deep networks. We use this approach to investigate learning in the context of vision tasks across a variety of state-of-the-art networks and provide insights into the learning behavior of these networks, including how task complexity affects layer-wise learning in deeper layers of networks.

View PDFchevron_right
Synaptic Noise as a Means of Implementing Weight-Perturbation Learning (Long Version)
Anthony Maidat

2005

Weight-perturbation (WP) algorithms for supervised and/or reinforcement learning offer improved biological plausibility over backpropagation,because of their reduced circuitry requirements for realization in neural hardware. All such algorithms use some form of information source — a means to compare weight changes with changes in output error — to adjust weights. This paper explores the hypothesis that biological synaptic noise might

View PDFchevron_right
Linear Backprop in non-linear networks
Mehrdad Yazdani

Backprop is the primary learning algorithm used in many machine learning algorithms. In practice, however, Backprop in deep neural networks is a highly sensitive learning algorithm and successful learning depends on numerous conditions and constraints. One set of constraints is to avoid weights that lead to saturated units. The motivation for avoiding unit saturation is that gradients vanish and as a result learning comes to a halt. Careful weight initialization and re-scaling schemes such as batch normalization ensure that input activity to the neuron is within the linear regime where gradients are not vanished and can flow. Here we investigate backpropagating error terms only linearly. That is, we ignore the saturation that arise by ensuring gradients always flow. We refer to this learning rule as Linear Backprop since in the backward pass the network appears to be linear. In addition to ensuring persistent gradient flow, Linear Backprop is also favorable when computation is expensive since gradients are never computed. Our early results suggest that learning with Linear Backprop is competitive with Backprop and saves expensive gradient computations.

View PDFchevron_right
Symmetry of backpropagation and chain rule
Andrzej Pacut

Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290), 2002

Gradient backpropagation, understood as a method of computing derivatives of composite functions, is commonly understood as a version of the chain rule. We show that this is not true, and both methods are in a sense opposite. While for the chain rule one needs derivatives with respect to all variables that influence a given intermediate variable, the backpropagation calls for derivatives of all variables that are influenced by the present variable. Knowing this, the derivation of the gradient for even complicated neural networks is almost trivial. In a matrix form, both methods differ in the order of matrix multiplication. Use of the chain rule is almost automatic since we all know it from math analysis education; use of the backpropagation could be as automatic if introduced in university math education as an equivalent alternative version of derivative calculation for composite functions.

View PDFchevron_right
Back Propagation is Sensitive to Initial Conditions
John Kolen

1990

This paper explores the effect of initial weight selection on feed-forward networks learning simple functions with the back-propagation technique. We first demonstrate, through the use of Monte Carlo techniques, that the magnitude of the initial condition vector (in weight space) is a very significant parameter in convergence time variability. In order to further understand this result, additional deterministic experiments were performed. The results of these experiments demonstrate the extreme sensitivity of back propagation to initial weight configuration.

View PDFchevron_right
Data Symmetries and Learning in Fully Connected Neural Networks
Fabio Anselmi

IEEE Access

Symmetries in the data and how they constrain the learned weights of modern deep networks is still an open problem. In this work we study the simple case of fully connected shallow non-linear neural networks and consider two types of symmetries: full dataset symmetries where the dataset X is mapped into itself by any transformation g, i.e. gX = X or single data point symmetries where gx = x, x ∈ X. We prove and experimentally confirm that symmetries in the data are directly inherited at the level of the network's learned weights and relate these findings with the common practice of data augmentation in modern machine learning. Finally, we show how symmetry constraints have a profound impact on the spectrum of the learned weights, an aspect of the so-called network implicit bias. INDEX TERMS Artificial neural networks, symmetry invariance, equivariance.

View PDFchevron_right

References (23)

  1. Abdel-Hamid et al. 2012] Abdel-Hamid, O.; Mohamed, A.; Jiang, H.; and Penn, G. 2012. Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4277-4280.
  2. Bengio et al. 2015] Bengio, Y.; Lee, D.-H.; Bornschein, J.; and Lin, Z. 2015. Towards biologically plausible deep learn- ing. arXiv preprint arXiv:1502.04156. [Bengio 2014] Bengio, Y. 2014. How auto-encoders could provide credit assignment in deep networks via target prop- agation. arXiv preprint arXiv:1407.7906. [Chinta and Tweed 2012] Chinta, L. V., and Tweed, D. B. 2012. Adaptive optimal control without weight transport. Neural computation 24(6):1487-1518.
  3. Coates, Ng, and Lee 2011] Coates, A.; Ng, A. Y.; and Lee, H. 2011. An analysis of single-layer networks in unsuper- vised feature learning. In International conference on artifi- cial intelligence and statistics, 215-223.
  4. Crick 1989] Crick, F. 1989. The recent excitement about neural networks. Nature 337(6203):129-132.
  5. Fanello et al. 2013] Fanello, S. R.; Ciliberto, C.; Santoro, M.; Natale, L.; Metta, G.; Rosasco, L.; and Odone, F. 2013. icub world: Friendly robots help building good vi- sion data-sets. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2013 IEEE Conference on, 700-705. IEEE.
  6. Fei-Fei, Fergus, and Perona 2007] Fei-Fei, L.; Fergus, R.; and Perona, P. 2007. Learning generative visual models from few training examples: An incremental bayesian ap- proach tested on 101 object categories. Computer Vision and Image Understanding 106(1):59-70. [Garofolo et al. ] Garofolo, J.; Lamel, L.; Fisher, W.; Fiscus, J.; Pallett, D.; Dahlgren, N.; and Zue, V. Timit acoustic- phonetic continuous speech corpus.
  7. Graves, Wayne, and Danihelka 2014] Graves, A.; Wayne, G.; and Danihelka, I. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401.
  8. Griffin, Holub, and Perona 2007] Griffin, G.; Holub, A.; and Perona, P. 2007. Caltech-256 object category dataset. [Grossberg 1987] Grossberg, S. 1987. Competitive learning: From interactive activation to adaptive resonance. Cognitive science 11(1):23-63.
  9. Hinton and McClelland 1988] Hinton, G. E., and McClel- land, J. L. 1988. Learning representations by recirculation. In Neural information processing systems, 358-366. New York: American Institute of Physics. [Hinton and Salakhutdinov 2006] Hinton, G. E., and Salakhutdinov, R. R. 2006. Reducing the dimensionality of data with neural networks. Science 313(5786):504-507.
  10. Hinton et al. 2012] Hinton, G.; Deng, L.; Yu, D.; Dahl, G. E.; Mohamed, A.-r.; Jaitly, N.; Senior, A.; Vanhoucke, V.; Nguyen, P.; Sainath, T. N.; et al. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE 29(6):82-97.
  11. Hornik, Stinchcombe, and White 1989] Hornik, K.; Stinch- combe, M.; and White, H. 1989. Multilayer feedforward networks are universal approximators. Neural networks 2(5):359-366.
  12. Huang et al. 2008] Huang, G. B.; Mattar, M.; Berg, T.; and Learned-Miller, E. 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained en- vironments. In Workshop on faces in real-life images: De- tection, alignment and recognition (ECCV). [Ioffe and Szegedy 2015] Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network train- ing by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.
  13. Krizhevsky, Sutskever, and Hinton 2012] Krizhevsky, A.; Sutskever, I.; and Hinton, G. 2012. ImageNet classification with deep convolutional neural networks. In Advances in neural information processing systems. [Krizhevsky 2009] Krizhevsky, A. 2009. Learning multiple layers of features from tiny images.
  14. Le Cun 1986] Le Cun, Y. 1986. Learning process in an asymmetric threshold network. In Disordered systems and biological organization. Springer. 233-240.
  15. LeCun, Cortes, and Burges ] LeCun, Y.; Cortes, C.; and Burges, C. J. The mnist database.
  16. Leibo, Liao, and Poggio 2014] Leibo, J. Z.; Liao, Q.; and Poggio, T. 2014. Subtasks of Unconstrained Face Recogni- tion. In International Joint Conference on Computer Vision, Imaging and Computer Graphics, VISIGRAPP. [Lillicrap et al. 2014] Lillicrap, T. P.; Cownden, D.; Tweed, D. B.; and Akerman, C. J. 2014. Random feedback weights support learning in deep neural networks. arXiv preprint arXiv:1411.0247.
  17. Mazzoni, Andersen, and Jordan 1991] Mazzoni, P.; Ander- sen, R. A.; and Jordan, M. I. 1991. A more biologically plausible learning rule for neural networks. Proceedings of the National Academy of Sciences 88(10):4433-4437.
  18. Mikolov et al. 2013] Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed represen- tations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NIPS), 3111-3119.
  19. Netzer et al. 2011] Netzer, Y.; Wang, T.; Coates, A.; Bis- sacco, A.; Wu, B.; and Ng, A. Y. 2011. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learn- ing, volume 2011, 5. [Nilsback and Zisserman 2006] Nilsback, M.-E., and Zisser- man, A. 2006. A visual vocabulary for flower classifica- tion. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. IEEE. [Nilsback and Zisserman 2008] Nilsback, M.-E., and Zisser- man, A. 2008. Automated flower classification over a large number of classes. In Computer Vision, Graphics & Image Processing, 2008. ICVGIP'08. Sixth Indian Conference on. IEEE.
  20. O'Reilly 1996] O'Reilly, R. C. 1996. Biologically plausi- ble error-driven learning using local activation differences: The generalized recirculation algorithm. Neural computa- tion 8(5):895-938.
  21. Pinto et al. 2011] Pinto, N.; Stone, Z.; Zickler, T.; and Cox, D. 2011. Scaling up biologically-inspired computer vi- sion: A case study in unconstrained face recognition on face- book. In Computer Vision and Pattern Recognition Work- shops (CVPRW), 2011 IEEE Computer Society Conference on, 35-42. IEEE. [Quattoni and Torralba 2009] Quattoni, A., and Torralba, A. 2009. Recognizing indoor scenes. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, 413-420. IEEE. [Riedmiller and Braun 1993] Riedmiller, M., and Braun, H. 1993. A direct adaptive method for faster backpropagation learning: The rprop algorithm. In Neural Networks, 1993., IEEE International Conference on, 586-591. IEEE.
  22. Rumelhart, Hinton, and Williams 1988] Rumelhart, D. E.; Hinton, G. E.; and Williams, R. J. 1988. Learning repre- sentations by back-propagating errors. Cognitive modeling. [Smolensky 1986] Smolensky, P. 1986. Information process- ing in dynamical systems: Foundations of harmony theory. [Stellwagen and Malenka 2006] Stellwagen, D., and Malenka, R. C. 2006. Synaptic scaling mediated by glial tnf-α. Nature 440(7087):1054-1059. [Taigman et al. 2014] Taigman, Y.; Yang, M.; Ranzato, M.; and Wolf, L. 2014. Deepface: Closing the gap to human- level performance in face verification. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, 1701-1708. IEEE. [Turrigiano and Nelson 2004] Turrigiano, G. G., and Nelson, S. B. 2004. Homeostatic plasticity in the developing nervous system. Nature Reviews Neuroscience. [Turrigiano 2008] Turrigiano, G. G. 2008. The self-tuning neuron: synaptic scaling of excitatory synapses. Cell 135(3):422-435.
  23. Vedaldi and Lenc 2015] Vedaldi, A., and Lenc, K. 2015. MatConvNet -Convolutional Neural Networks for MAT- LAB [Zamanidoost et al. 2015] Zamanidoost, E.; Bayat, F. M.; Strukov, D.; and Kataeva, I. 2015. Manhattan rule training for memristive crossbar circuit pattern classifiers. [Zamanidoost et al. 2015] Zamanidoost, E.; Bayat, F. M.; Strukov, D.; and Kataeva, I. 2015. Manhattan rule training for memristive crossbar circuit pattern classifiers.

Related papers

Backpropagation without weight transport
John Kolen

1994

In backpropagation, connection weights are used to both compute node activations and error gradient for hidden units. Grossberg (1987) has argued that the dual use of the same synaptic connections (weight transport) constitutes a bidirectional flow of information through synapses, which is biologically implausable. In this paper we formally and empirically demonstrate the feasibility of an architecture equivalent to backpropagation, but without the assumption of weight transport. Through coordinated training with weight decay, a reciprocal layer of weights evolves into a copy of the forward connections and acts as the conduit for backward flowing corrective information. Examination of the networks trained with dual weights suggests that functional synchronization, and not weight synchronization, is crucial to the operation of backpropagation methods

View PDFchevron_right
Weight Changes for Learning Mechanisms in Two- Term Back-Propagation Network
Siti Mariyam Shamsuddin
View PDFchevron_right
Learning in the machine: The symmetries of the deep learning channel
Peter Sadowski

Neural Networks, 2017

In a physical neural system, learning rules must be local both in space and time. In order for learning to occur, nonlocal information must be communicated to the deep synapses through a communication channel, the deep learning channel. We identify several possible architectures for this learning channel (Bidirectional, Conjoined, Twin, Distinct) and six symmetry challenges: 1) symmetry of architectures; 2) symmetry of weights; 3) symmetry of neurons; 4) symmetry of derivatives; 5) symmetry of processing; and 6) symmetry of learning rules. Random backpropagation (RBP) addresses the second and third symmetry, and some of its variations, such as skipped RBP (SRBP) address the first and the fourth symmetry. Here we address the last two desirable symmetries showing through simulations that they can be achieved and that the learning channel is particularly robust to symmetry variations. Specifically, random backpropagation and its variations can be performed with the same non-linear neurons used in the main input-output forward channel, and the connections in the learning channel can be adapted using the same algorithm used in the forward channel, removing the need for any specialized hardware in the learning channel. Finally, we provide mathematical results in simple cases showing that the learning equations in the forward and backward channels converge to fixed points, for almost any initial conditions. In symmetric architectures, if the weights in both channels are small at initialization, adaptation in both channels leads to weights that are essentially symmetric during and after learning. Biological connections are discussed.

View PDFchevron_right
Learning Invariant Weights in Neural Networks
Tycho van der Ouderaa

Cornell University - arXiv, 2022

Assumptions about invariances or symmetries in data can significantly increase the predictive power of statistical models. Many commonly used machine learning models are constraint to respect certain symmetries, such as translation equivariance in convolutional neural networks, and incorporating other symmetry types is actively being studied. Yet, learning invariances from the data itself remains an open research problem. It has been shown that the marginal likelihood offers a principled way to learn invariances in Gaussian Processes. We propose a weight-space equivalent to this approach, by minimizing a lower bound on the marginal likelihood to learn invariances in neural networks, resulting in naturally higher performing models. Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012. Sebastian W Ober and Laurence Aitchison. Global inducing point variational posteriors for bayesian neural networks and deep gaussian processes. arXiv preprint arXiv:2005.08140, 2020.

View PDFchevron_right
Uniqueness of weights for neural networks
eduardo sontag

1993

View PDFchevron_right

Related topics

  • Computer Science
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved