Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Methodology
Spectral Spacing Statistics in RMT
Results
Conclusion and Future Work
References
All Topics
Physics
Mathematical Physics

Appearance of Random Matrix Theory in deep learning

Profile image of rangarirai mbizirangarirai mbizi

2022, Physica A: Statistical Mechanics and its Applications

https://doi.org/10.1016/J.PHYSA.2021.126742
visibility

…

description

33 pages

link

1 file

Abstract

We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover agreement with Gaussian Orthogonal Ensemble statistics across several network architectures and datasets. These results shed new light on the applicability of Random Matrix Theory to modelling neural networks and suggest a role for it in the study of loss surfaces in deep learning.

Figures (14)
Figure 1: Comparison of different global spectral statistics (spectral densities). (a) We show actual GOE data to demonstrate the form of the Wigner semicircle. (b) Hessian of cross entropy loss for MLP on MNIST. (c) Hessian of cross entropy loss for logistic regression on MNIST. Note the log-scale on the y-axis. A few outliers have been clipped from logistic regression to aid visualisation.
Figure 1: Comparison of different global spectral statistics (spectral densities). (a) We show actual GOE data to demonstrate the form of the Wigner semicircle. (b) Hessian of cross entropy loss for MLP on MNIST. (c) Hessian of cross entropy loss for logistic regression on MNIST. Note the log-scale on the y-axis. A few outliers have been clipped from logistic regression to aid visualisation.
Figure 2: The density of the Wigner surmise. not just the GOE or Wigner random matrices.
Figure 2: The density of the Wigner surmise. not just the GOE or Wigner random matrices.
Figure 3: Spacing distributions for the Hessian of a logistic regression trained Resnet-34 embeddings of CIFAR10. Hessians computed over the test set. We display results as histograms of data along with a plot of the Wigner (or the Wigner-like) surmise density. We make a few practical adjustments to the plots. Spacing ratios are truncated above some value, as the presence of a few extreme outliers makes visualisation difficult. We choose a cut-off at 10. Note that around 0.985 of the mass of the Wigner-like surmise is below 10, so this is a reasonable adjustment. The hessians have degenerate spectra. The Wigner surmise is not a good fit to the observed unfolded spectra if the zero eigenvalues are retained. Imposing a lower cut-off of 10~?° in magnitude is sufficient to obtain agreement with Wigner.'? This is below the machine precision, so these omitted eigenvalues are indistinguishable from 0. 6.1. MNIST and MLPs
Figure 3: Spacing distributions for the Hessian of a logistic regression trained Resnet-34 embeddings of CIFAR10. Hessians computed over the test set. We display results as histograms of data along with a plot of the Wigner (or the Wigner-like) surmise density. We make a few practical adjustments to the plots. Spacing ratios are truncated above some value, as the presence of a few extreme outliers makes visualisation difficult. We choose a cut-off at 10. Note that around 0.985 of the mass of the Wigner-like surmise is below 10, so this is a reasonable adjustment. The hessians have degenerate spectra. The Wigner surmise is not a good fit to the observed unfolded spectra if the zero eigenvalues are retained. Imposing a lower cut-off of 10~?° in magnitude is sufficient to obtain agreement with Wigner.'? This is below the machine precision, so these omitted eigenvalues are indistinguishable from 0. 6.1. MNIST and MLPs
Figure 4: Spacing distributions for the Hessian of a 3-hidden-layer MLP trained on MNIST. Hessians computed over the test set. points, while spacing ratios were computed in batches and over the entire dataset. We observe a striking level of agreement between the observed spectra and the GO] FH. There was no discernible difference between the train and test conditions, nor between batch and full dataset conditions, nor between trained and untrained models. Note that the presence of GOE statistics for the untrained models is not a foregone conclusion. Of course, the weights of the model are indeed random Gaussian, but the Hessian is still a function of the data set, so it is not the case the Hessian eigenvalue statistics are bound to be GOE a pr iori. Overall, the very close agreement between Random Matrix Theory predictions and our observations for several different architectures, model sizes and datasets demonstrates a clear presence of RMT statistics in neural networks.
Figure 4: Spacing distributions for the Hessian of a 3-hidden-layer MLP trained on MNIST. Hessians computed over the test set. points, while spacing ratios were computed in batches and over the entire dataset. We observe a striking level of agreement between the observed spectra and the GO] FH. There was no discernible difference between the train and test conditions, nor between batch and full dataset conditions, nor between trained and untrained models. Note that the presence of GOE statistics for the untrained models is not a foregone conclusion. Of course, the weights of the model are indeed random Gaussian, but the Hessian is still a function of the data set, so it is not the case the Hessian eigenvalue statistics are bound to be GOE a pr iori. Overall, the very close agreement between Random Matrix Theory predictions and our observations for several different architectures, model sizes and datasets demonstrates a clear presence of RMT statistics in neural networks.
Figure 5: Spectral densities of Gaussian process Hessians with various kernel choices. All use k’(0) = 1. The dimension is 300 in all cases except (d), in which the Hessian is padded to 400 dimensions with zeros. All histograms are produced with 100 independent Hessian samples. « = 100 degenerate directions. + = 20 outliers
Figure 5: Spectral densities of Gaussian process Hessians with various kernel choices. All use k’(0) = 1. The dimension is 300 in all cases except (d), in which the Hessian is padded to 400 dimensions with zeros. All histograms are produced with 100 independent Hessian samples. « = 100 degenerate directions. + = 20 outliers
Figure 6: Consecutive spacing ratios of Gaussian process Hessians with various kernel choices. All use k’(0) = 1. The dimension is 300 in all cases except (d), in which the Hessian is padded to 400 dimensions with zeros. * = 100 degenerate directions. | = 20 outliers.
Figure 6: Consecutive spacing ratios of Gaussian process Hessians with various kernel choices. All use k’(0) = 1. The dimension is 300 in all cases except (d), in which the Hessian is padded to 400 dimensions with zeros. * = 100 degenerate directions. | = 20 outliers.
Figure 7: Spectral statistics for the Hessian of a CNN trained on CIFAR10. Hessians computed over batches of size 64 on the test set.
Figure 7: Spectral statistics for the Hessian of a CNN trained on CIFAR10. Hessians computed over batches of size 64 on the test set.
Figure 8: Spectral statistics for the Hessian of an MLP trained on the Bike dataset. Hessians computed over batches of size 64 on the test set.
Figure 8: Spectral statistics for the Hessian of an MLP trained on the Bike dataset. Hessians computed over batches of size 64 on the test set.
Figure 9: Spectral statistics for the Gauss-Newton matrix of an MLP trained on the Bike dataset. Matrices computed over batches of size 64 on the test set. 7. Conclusion and future work
Figure 9: Spectral statistics for the Gauss-Newton matrix of an MLP trained on the Bike dataset. Matrices computed over batches of size 64 on the test set. 7. Conclusion and future work
Figure A.10: Unfolded spacings for the Hessian of a logistic regression trained on MNIST. Hessian computed batches of size 64 of the training and test datasets. Figure A.14 compares the effect of degeneracy on unfolded spacings in each of the 3 cases considered. We see that the logistic MNIST models (trained and untrained) have a much greater level of degeneracy, whereas the CIFAR10-Resnet34 spectra clearly have GO] F} spacings even without any cut-off. Figures A.11—A.13 show further unfolded spacing and spacing ratio results like those in the main text.
Figure A.10: Unfolded spacings for the Hessian of a logistic regression trained on MNIST. Hessian computed batches of size 64 of the training and test datasets. Figure A.14 compares the effect of degeneracy on unfolded spacings in each of the 3 cases considered. We see that the logistic MNIST models (trained and untrained) have a much greater level of degeneracy, whereas the CIFAR10-Resnet34 spectra clearly have GO] F} spacings even without any cut-off. Figures A.11—A.13 show further unfolded spacing and spacing ratio results like those in the main text.
Figure A.11: Consecutive spacing ratios for the Hessian of a logistic regression trained on MNIST. Hessian computed batches of size 64 of the training and test sets, and over the whole train and test sets.
Figure A.11: Consecutive spacing ratios for the Hessian of a logistic regression trained on MNIST. Hessian computed batches of size 64 of the training and test sets, and over the whole train and test sets.
Figure A.12: Unfolded spacings for the Hessian of a randomly initialised logistic regression for MNIST. Hessian computed batches of size 64 of the training and test datasets.
Figure A.12: Unfolded spacings for the Hessian of a randomly initialised logistic regression for MNIST. Hessian computed batches of size 64 of the training and test datasets.
Figure A.13: Consecutive spacing ratios for the Hessian of a randomly initialised logistic regression for MNIST. Hessian computed batches of size 64 of the training and test sets, and over the whole train and test sets.
Figure A.13: Consecutive spacing ratios for the Hessian of a randomly initialised logistic regression for MNIST. Hessian computed batches of size 64 of the training and test sets, and over the whole train and test sets.

Related papers

Spectral complexity of deep neural networks
Simmaco Di Lillo

arXiv (Cornell University), 2024

It is well-known that randomly initialized, push-forward, fully-connected neural networks weakly converge to isotropic Gaussian processes, in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting fields to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum, and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or highdisorder; we show how this classification highlights a number of distinct features for standard activation functions, and in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations.

View PDFchevron_right
On the modeling of error functions as high dimensional landscapes for weight initialization in learning networks
Gopinath Mahale

2016 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation (SAMOS), 2016

Next generation deep neural networks for classification hosted on embedded platforms will rely on fast, efficient, and accurate learning algorithms. Initialization of weights in learning networks has a great impact on the classification accuracy. In this paper we focus on deriving good initial weights by modeling the error function of a deep neural network as a high-dimensional landscape. We observe that due to the inherent complexity in its algebraic structure, such an error function may conform to general results of the statistics of large systems. To this end we apply some results from Random Matrix Theory to analyse these functions. We model the error function in terms of a Hamiltonian in N-dimensions and derive some theoretical results about its general behavior. These results are further used to make better initial guesses of weights for the learning algorithm.

View PDFchevron_right
On the Variance of the Fisher Information for Deep Learning
Alexander Soen

2021

The Fisher information matrix (FIM) has been applied to the realm of deep learning. It is closely related to the loss landscape, the variance of the parameters, second order optimization, and deep learning theory. However, the exact FIM is either unavailable in closed form or too expensive to compute. In practice, it is almost always estimated based on empirical samples. We investigate two such estimators based on two equivalent representations of the FIM – both unbiased and consistent with respect to the underlying “true” FIM. Their estimation quality is characterized by their variance given in closed form. We bound their variances and analyze how the parametric structure of a deep neural network can impact the variance. We discuss the meaning of this variance measure and our bounds in the context of deep learning.

View PDFchevron_right
Harnessing neural networks: A random matrix approach
Cosme Louart

2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017

This article proposes an original approach to the performance understanding of large dimensional neural networks. In this preliminary study, we study a single hidden layer feed-forward network with random input connections (also called extreme learning machine) which performs a simple regression task. By means of a new random matrix result, we prove that, as the size and cardinality of the input data and the number of neurons grow large, the network performance is asymptotically deterministic. This entails a better comprehension of the effects of the hyper-parameters (activation function, number of neurons, etc.) under this simple setting, thereby paving the path to the harnessing of more involved structures.

View PDFchevron_right
Chapter Title Deep Learning with Random Neural Networks Deep Learning with Random Neural Networks
Erol Gelenbe, Yonghua Yin

This paper develops multi-layer classifiers and auto-encoders based on the Random Neural Network. Our motivation is to build robust classifiers that can be used in systems applications such as Cloud management for the accurate detection of states that can lead to failures. Using an idea concerning some to soma interactions between natural neuronal cells, we discuss a basic building block constructed of clusters of densely packet cells whose mathematical properties are based on G-Networks and the Random Neural Network. These mathematical properties lead to a transfer function that can be exploited for large arrays of cells. Based on this mathematical structure we build multi-layer networks. In order to evaluate the level of classification accuracy that can be achieved, we test these auto-encoders and classifiers on a widely used standard database of handwritten characters. Abstract. This paper develops multi-layer classifiers and auto-encoders based on the Random Neural Network. Our motivation is to build robust classifiers that can be used in systems applications such as Cloud management for the accurate detection of states that can lead to failures. Using an idea concerning some to soma interactions between natural neuronal cells, we discuss a basic building block constructed of clusters of densely packet cells whose mathematical properties are based on G-Networks and the Random Neural Network. These mathematical properties lead to a transfer function that can be exploited for large arrays of cells. Based on this mathematical structure we build multi-layer networks. In order to evaluate the level of classification accuracy that can be achieved, we test these auto-encoders and classifiers on a widely used standard database of handwritten characters. AQ1

View PDFchevron_right
Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
Tiancong Chen

Proceedings of the 2020 SIAM International Conference on Data Mining, 2020

While stochastic gradient descent (SGD) and variants have been surprisingly successful for training deep nets, several aspects of the optimization dynamics and generalization are still not well understood. In this paper, we present new empirical observations and theoretical results on both the optimization dynamics and generalization behavior of SGD for deep nets based on the Hessian of the training loss and associated quantities. We consider three specific research questions: (1) what is the relationship between the Hessian of the loss and the second moment of stochastic gradients (SGs)? (2) how can we characterize the stochastic optimization dynamics of SGD with fixed step sizes based on the first and second moments of SGs? and (3) how can we characterize a scale-invariant generalization bound of deep nets based on the Hessian of the loss? Throughout the paper, we support theoretical results with empirical observations, with experiments on synthetic data, MNIST, and CIFAR-10, with different batch sizes, and with different difficulty levels by synthetically adding random labels.

View PDFchevron_right
Analytic Insights into Structure and Rank of Neural Network Hessian Maps
Sidak Preet Singh 20EAREC067

ArXiv, 2021

The Hessian of a neural network captures parameter interactions through secondorder derivatives of the loss. It is a fundamental object of study, closely tied to various problems in deep learning, including model design, optimization, and generalization. Most prior work has been empirical, typically focusing on low-rank approximations and heuristics that are blind to the network structure. In contrast, we develop theoretical tools to analyze the range of the Hessian map, which provide us with a precise understanding of its rank deficiency and the structural reasons behind it. This yields exact formulas and tight upper bounds for the Hessian rank of deep linear networks — allowing for an elegant interpretation in terms of rank deficiency. Moreover, we demonstrate that our bounds remain faithful as an estimate of the numerical Hessian rank, for a larger class of models such as rectified and hyperbolic tangent networks. Further, we also investigate the implications of model architectur...

View PDFchevron_right
Predicting the outputs of finite networks trained with noisy gradients
Oded Ben-David

2020

A recent line of studies has focused on the infinite width limit of deep neural networks (DNNs) where, under a certain deterministic training protocol, the DNN outputs are related to a Gaussian Process (GP) known as the Neural Tangent Kernel (NTK). However, finite-width DNNs differ from GPs quantitatively and for CNNs the difference may be qualitative. Here we present a DNN training protocol involving noise whose outcome is mappable to a certain non-Gaussian stochastic process. An analytical framework is then introduced to analyze this resulting non-Gaussian process, whose deviation from a GP is controlled by the finite width. Our work extends upon previous relations between DNNs and GPs in several ways: (a) In the infinite width limit, it establishes a mapping between DNNs and a GP different from the NTK. (b) It allows computing analytically the general form of the finite width correction (FWC) for DNNs with arbitrary activation functions and depth and further provides insight on t...

View PDFchevron_right
Predicting the outputs of finite deep neural networks trained with noisy gradients
Oded Ben-David

2020

A recent line of works studied wide deep neural networks (DNNs) by approximating them as Gaussian Processes (GPs). A DNN trained with gradient flow was shown to map to a GP governed by the Neural Tangent Kernel (NTK), whereas earlier works showed that a DNN with an i.i.d. prior over its weights maps to the so-called Neural Network Gaussian Process (NNGP). Here we consider a DNN training protocol, involving noise, weight decay and finite width, whose outcome corresponds to a certain non-Gaussian stochastic process. An analytical framework is then introduced to analyze this non-Gaussian process, whose deviation from a GP is controlled by the finite width. Our contribution is three-fold: (i) In the infinite width limit, we establish a correspondence between DNNs trained with noisy gradients and the NNGP, not the NTK. (ii) We provide a general analytical form for the finite width correction (FWC) for DNNs with arbitrary activation functions and depth and use it to predict the outputs of...

View PDFchevron_right
Normal approximation of Random Gaussian Neural Networks
Daniela De Canditiis

arXiv (Cornell University), 2023

In this paper we provide explicit upper bounds on some distances between the (law of the) output of a random Gaussian neural network and (the law of) a random Gaussian vector. Our main results concern deep random Gaussian neural networks, with a rather general activation function. The upper bounds show how the widths of the layers, the activation function and other architecture parameters affect the Gaussian approximation of the output. Our techniques, relying on Stein's method and integration by parts formulas for the Gaussian law, yield estimates on distances which are indeed integral probability metrics, and include the convex distance. This latter metric is defined by testing against indicator functions of measurable convex sets, and so allow for accurate estimates of the probability that the output is localized in some region of the space. Such estimates have a significant interest both from a practitioner's and a theorist's perspective.

View PDFchevron_right

References (86)

  1. Sherif M Abuelenin and Adel Y Abul-Magd. Effect of unfolding on the spectral statistics of adjacency matrices of complex networks. Procedia Computer Science, 12:69-74, 2012.
  2. Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pages 74-84. PMLR, 2020.
  3. Robert J Adler and Jonathan E Taylor. Random fields and geometry. Springer Science & Business Media, 2009.
  4. Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky. Spin-glass models of neural networks. Physical Review A, 32(2):1007, 1985.
  5. YY Atas, E Bogomolny, O Giraud, and G Roux. Distribution of the ratio of consecutive level spacings in random matrix ensembles. Physical review letters, 110(8):084101, 2013.
  6. Antonio Auffinger, Gérard Ben Arous, and Jiří Černỳ. Random matrices and complexity of spin glasses. Communications on Pure and Applied Mathematics, 66(2):165-201, 2013.
  7. Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Denny Wu, and Tianzong Zhang. Generalization of two-layer neural net-works: An asymptotic viewpoint. risk, 1(1.5):2-0, 2020.
  8. Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoen- holz, Jascha Sohl-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 2020.
  9. Nicholas P Baskerville, Jonathan P Keating, Francesco Mezzadri, and Joseph Najnudel. The loss surfaces of neural networks with general activation functions. Journal of Statistical Mechanics: Theory and Experiment, 2021(6):064001, 2021.
  10. Nicholas P Baskerville, Jonathan P Keating, Francesco Mezzadri, and Joseph Najnudel. A spin-glass model for the loss surfaces of generative adversarial networks. arXiv preprint arXiv:2101.02524, 2021.
  11. Carlo WJ Beenakker. Random-matrix theory of quantum transport. Reviews of modern physics, 69(3):731, 1997.
  12. Lucas Benigni and Sandrine Péché. Eigenvalue distribution of nonlinear models of random matrices. arXiv preprint arXiv:1904.03090, 2019.
  13. Michael V Berry and Marko Robnik. Semiclassical level spacings when regular and chaotic orbits coexist. Journal of Physics A: Mathematical and General, 17(12):2413, 1984.
  14. Michael V Berry et al. Quantum chaology. Proc. Roy. Soc. London A, 413:183-198, 1987.
  15. Michael Victor Berry and Michael Tabor. Level clustering in the regular spectrum. Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences, 356(1686):375-394, 1977.
  16. Oriol Bohigas. Random matrix theories and chaotic dynamics. Technical report, Paris-11 Univ., 1991.
  17. Léon Bottou. Stochastic Gradient Descent Tricks, pages 421-436. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642- 35289-8. doi: 10.1007/978-3-642-35289-8 25. URL https://doi.org/ 10.1007/978-3-642-35289-8_25.
  18. Joël Bun, Jean-Philippe Bouchaud, and Marc Potters. Cleaning large correlation matrices: tools from random matrix theory. Physics Reports, 666:1-109, 2017.
  19. Tianle Cai, Ruiqi Gao, Jikai Hou, Siyu Chen, Dong Wang, Di He, Zhihua Zhang, and Liwei Wang. Gram-gauss-newton method: Learning over- parameterized neural networks for regression problems. arXiv preprint arXiv:1905.11675, 2019.
  20. Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Reviews of Modern Physics, 91(4):045002, 2019.
  21. Pratik Chaudhari and Stefano Soatto. On the energy landscape of deep networks. arXiv preprint arXiv:1511.06485, 2015.
  22. Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, pages 192-204, 2015.
  23. Anna Choromanska, Yann LeCun, and Gérard Ben Arous. Open problem: The landscape of the loss surfaces of multilayer networks. In Conference on Learning Theory, pages 1756-1760, 2015.
  24. Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in neural information processing systems, pages 2933-2941, 2014.
  25. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
  26. Oussama Dhifallah and Yue M Lu. A precise performance analysis of learning with random features. arXiv preprint arXiv:2008.11904, 2020.
  27. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Jul):2121-2159, 2011.
  28. Konstantin Efetov. Supersymmetry in disorder and chaos. Cambridge university press, 1999.
  29. Yan V Fyodorov. Complexity of random energy landscapes, glass transi- tion, and absolute value of the spectral determinant of random matrices. Physical review letters, 92(24):240601, 2004.
  30. Yan V Fyodorov and Pierre Le Doussal. Topology trivialization and large deviations for the minimum in the simplest random optimization. Journal of Statistical Physics, 154(1):466-490, 2014.
  31. Yan V Fyodorov and Pierre Le Doussal. Hessian spectrum at the global minimum of high-dimensional random landscapes. Journal of Physics A: Mathematical and Theoretical, 51(47):474002, 2018.
  32. Yan V Fyodorov and Ian Williams. Replica symmetry breaking condition exposed by random matrix calculation of landscape complexity. Journal of Statistical Physics, 129(5-6):1081-1116, 2007.
  33. Marylou Gabrié. Mean-field inference methods for neural networks. Journal of Physics A: Mathematical and Theoretical, 53(22):223002, 2020.
  34. Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general, 21(1):271, 1988.
  35. Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning, pages 3452-3462. PMLR, 2020.
  36. Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian eigenvalue density. arXiv preprint arXiv:1901.10159, 2019.
  37. Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The gaussian equivalence of genera- tive models for learning with shallow neural networks. arXiv preprint arXiv:2006.14709, 2020.
  38. Diego Granziol. Beyond random matrix theory for deep networks. arXiv preprint arXiv:2006.07721, 2020.
  39. Diego Granziol, Timur Garipov, Dmitry Vetrov, Stefan Zohren, Stephen Roberts, and Andrew Gordon Wilson. Towards understanding the true loss surface of deep neural networks using random matrix theory and iterative spectral methods. 2019.
  40. Diego Granziol, Xingchen Wan, Timur Garipov, Dmitry Vetrov, and Stephen Roberts. MLRG deep curvature. arXiv preprint arXiv:1912.09656, 2019.
  41. Diego Granziol, Timur Garipov, Dmitry Vetrov, Stefan Zohren, Stephen Roberts, and Andrew Gordon Wilson. Towards understanding the true loss surface of deep neural networks using random matrix the- ory and iterative spectral methods. https://openreview.net/forum? id=H1gza2NtwH, 2020.
  42. Thomas Guhr, Axel Müller-Groeling, and Hans A Weidenmüller. Random-matrix theories in quantum physics: common concepts. Physics Reports, 299(4-6):189-425, 1998.
  43. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
  44. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
  45. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.07572, 2018.
  46. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  47. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
  48. Yann LeCun. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
  49. Meng Heng Loke and Torleif Dahlin. A comparison of the gauss-newton and quasi-newton methods in resistivity imaging inversion. Journal of applied geophysics, 49(3):149-162, 2002.
  50. Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Capturing the learning curves of generic features maps for realistic data sets with a teacher- student model. arXiv preprint arXiv:2102.08127, 2021.
  51. Antoine Maillard, Gérard Ben Arous, and Giulio Biroli. Landscape com- plexity for the empirical risk of generalized linear models. In Mathematical and Scientific Machine Learning, pages 287-327. PMLR, 2020.
  52. James Martens. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193, 2014.
  53. James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International conference on machine learning, pages 2408-2417, 2015.
  54. James Martens and Ilya Sutskever. Training deep and recurrent networks with Hessian-free optimization. In Neural networks: Tricks of the trade, pages 479-535. Springer, 2012.
  55. Madan Lal Mehta. Random matrices. Elsevier, 2004.
  56. Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 2019.
  57. Gérard Meurant and Zdeněk Strakoš. The Lanczos and conjugate gradient algorithms in finite precision arithmetic. Acta Numerica, 15:471-542, 2006.
  58. Marc Mezard and Andrea Montanari. Information, physics, and compu- tation. Oxford University Press, 2009.
  59. Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size. arXiv preprint arXiv:1811.07062, 2018.
  60. Vardan Papyan. Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians. arXiv preprint arXiv:1901.08244, 2019.
  61. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in Pytorch. 2017.
  62. Barak A Pearlmutter. Fast exact multiplication by the Hessian. Neural computation, 6(1):147-160, 1994.
  63. Jeffrey Pennington and Yasaman Bahri. Geometry of neural network loss surfaces via random matrix theory. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2798- 2806. JMLR. org, 2017.
  64. Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of a single-hidden-layer neural network. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 5410-5419. Curran Associates, Inc., 2018.
  65. Jeffrey Pennington and Pratik Worah. Nonlinear random matrix the- ory for deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124005, 2019.
  66. PyTorch. Resnet. https://pytorch.org/hub/pytorch_vision_ resnet, 2021. Accessed: 2021-05-03.
  67. Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. arXiv preprint arXiv:2106.10165, 2021.
  68. Valentina Ros, Gerard Ben Arous, Giulio Biroli, and Chiara Cammarota. Complex energy landscapes in spiked-tensor and simple glassy mod- els: Ruggedness, arrangements of local minima, and phase transitions. Physical Review X, 9(1):011003, 2019.
  69. Levent Sagun, V Ugur Guney, Gerard Ben Arous, and Yann Le- Cun. Explorations on high dimensional landscapes. arXiv preprint arXiv:1412.6615, 2014.
  70. Levent Sagun, Léon Bottou, and Yann LeCun. Eigenvalues of the Hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476, 2016.
  71. Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the Hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.04454, 2017.
  72. Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Flo- rent Krzakala, and Lenka Zdeborová. Who is afraid of big bad minima? analysis of gradient-flow in spiked matrix-tensor mod- els.
  73. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural In- formation Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ fbad540b2f3b5638a9be9aa6a4d8e450-Paper.pdf.
  74. Torsten Scholak. unfoldr. https://github.com/tscholak/unfoldr, 2015. Accessed: 2020-10-30.
  75. Torsten Scholak, Thomas Wellens, and Andreas Buchleitner. Spectral backbone of excitation transport in ultracold rydberg gases. Physical Review A, 90(6):063415, 2014.
  76. Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806-813, 2014.
  77. Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Soc., 2012.
  78. Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Wein- berger, and Andrew Gordon Wilson. Exact gaussian processes on a million data points. Advances in Neural Information Processing Systems, 32:14648-14659, 2019.
  79. HA Weidenmuller and GE Mitchell. Random matrices and chaos in nuclear physics. arXiv preprint arXiv:0807.1070, 2008.
  80. Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: Thresholds and algorithms. Advances in Physics, 65(5):453-552, 2016.
  81. Appendix B. Experimental details 3. 100 neurons to 100 neurons.
  82. 100 neurons to 10 output logits. Logistic regression on ResNet features (CIFAR10)
  83. Fully connection layer from 400 to 120.
  84. Fully connection layer from 120 to 84.
  85. Fully connection layer from 84 to output 10 logits. MLP (CIFAR10)
  86. 50 neurons to 1 regression output.

Related papers

On random matrices arising in deep neural networks: General I.I.D. case
Leonid Pastur

Random Matrices: Theory and Applications

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gauss...

View PDFchevron_right
A random matrix approach to neural networks
Cosme Louart

The Annals of Applied Probability

This article studies the Gram random matrix model G = 1 T Σ T Σ, Σ = σ(W X), classically found in the analysis of random feature maps and random neural networks, where X = [x1, . . . , xT ] ∈ R p×T is a (data) matrix of bounded norm, W ∈ R n×p is a matrix of independent zero-mean unit variance entries, and σ : R → R is a Lipschitz continuous (activation) function -σ(W X) being understood entrywise. By means of a key concentration of measure lemma arising from non-asymptotic random matrix arguments, we prove that, as n, p, T grow large at the same rate, the resolvent Q = (G + γIT ) -1 , for γ > 0, has a similar behavior as that met in sample covariance matrix models, involving notably the moment Φ = T n E[G], which provides in passing a deterministic equivalent for the empirical spectral measure of G. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters. * Couillet's work is supported by the ANR Project RMT4GRAPH (ANR-14-CE28-0006).

View PDFchevron_right
On the Power-Law Hessian Spectrums in Deep Learning
Yunfeng Cai

arXiv (Cornell University), 2022

It is well-known that the Hessian of deep loss landscape matters to optimization, generalization, and even robustness of deep learning. Recent works empirically discovered that the Hessian spectrum in deep learning has a two-component structure that consists of a small number of large eigenvalues and a large number of nearly-zero eigenvalues. However, the theoretical mechanism or the mathematical behind the Hessian spectrum is still largely under-explored. To the best of our knowledge, we are the first to demonstrate that the Hessian spectrums of welltrained deep neural networks exhibit simple power-law structures. Inspired by the statistical physical theories and the spectral analysis of natural proteins, we provide a maximum-entropy theoretical interpretation for explaining why the power-law structure exist and suggest a spectral parallel between protein evolution and training of deep neural networks. By conducing extensive experiments, we further use the power-law spectral framework as a useful tool to explore multiple novel behaviors of deep learning.

View PDFchevron_right
A Random Matrix and Concentration Inequalities Framework for Neural Networks Analysis
Cosme Louart

2018

This article provides a theoretical analysis of the asymptotic performance of a regression or classification task performed by a simple random neural network. This result is obtained by leveraging a new framework at the crossroads between random matrix theory and the concentration of measure theory. This approach is of utmost interest for neural network analysis at large in that it naturally dismisses the difficulty induced by the non-linear activation functions, so long that these are Lipschitz functions. As an application, we provide formulas for the limiting law of the random neural network output and compare them conclusively to those obtained practically on handwritten digits databases.

View PDFchevron_right
High-dimensional dynamics of generalization error in neural networks
Haim Sompolinsky

Neural Networks, 2020

We perform an average case analysis of the generalization dynamics of large neural networks trained using gradient descent. We study the practically-relevant "high-dimensional" regime where the number of free parameters in the network is on the order of or even larger than the number of examples in the dataset. Using random matrix theory and exact solutions in linear models, we derive the generalization error and training error dynamics of learning and analyze how they depend on the dimensionality of data and signal to noise ratio of the learning problem. We find that the dynamics of gradient descent learning naturally protect against overtraining and overfitting in large networks. Overtraining is worst at intermediate network sizes, when the effective number of free parameters equals the number of samples, and thus can be reduced by making a network smaller or larger. Additionally, in the high-dimensional regime, low generalization error requires starting with small initial weights. We then turn to non-linear neural networks, and show that making networks very large does not harm their generalization performance. On the contrary, it can in fact reduce overtraining, even without early stopping or regularization of any sort. We identify two novel phenomena underlying this behavior in overcomplete models: first, there is a frozen subspace of the weights in which no learning occurs under gradient descent; and second, the statistical properties of the high-dimensional regime yield better-conditioned input correlations which protect against overtraining. We demonstrate that naive application of worst-case theories such as Rademacher complexity are inaccurate in predicting the generalization performance of deep neural networks, and derive an alternative bound which incorporates the frozen subspace and conditioning effects and qualitatively matches the behavior observed in simulation.

View PDFchevron_right

Related topics

  • Mathematics
  • Mathematical Physics
  • Computer Science
  • Artificial Intelligence
  • Physics
  • Quantum Physics
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved