Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Background and Notation
Main Results
Conclusions and Future Work
References
All Topics
Computer Science

Spectral complexity of deep neural networks

Profile image of Simmaco Di LilloSimmaco Di Lillo

2024, arXiv (Cornell University)

https://doi.org/10.48550/ARXIV.2405.09541
visibility

…

description

31 pages

link

1 file

Abstract

It is well-known that randomly initialized, push-forward, fully-connected neural networks weakly converge to isotropic Gaussian processes, in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting fields to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum, and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or highdisorder; we show how this classification highlights a number of distinct features for standard activation functions, and in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations.

Related papers

Large-width functional asymptotics for deep Gaussian neural networks
Daniele Bracale

2021

In this paper, we consider fully-connected feed-forward deep neural networks where weights and biases are independent and identically distributed according to Gaussian distributions. Extending previous results (Matthews et al., 2018a;b; Yang, 2019) we adopt a function-space perspective, i.e. we look at neural networks as infinite-dimensional random elements on the input space R . Under suitable assumptions on the activation function we show that: i) a network defines a continuous stochastic process on the input space R ; ii) a network with re-scaled weights converges weakly to a continuous Gaussian Process in the large-width limit; iii) the limiting Gaussian Process has almost surely locally γ-Hölder continuous paths, for 0 < γ < 1. Our results contribute to recent theoretical studies on the interplay between infinitely-wide deep neural networks and Gaussian Processes by establishing weak convergence in function-space with respect to a stronger metric.

View PDFchevron_right
Non-asymptotic approximations of neural networks by Gaussian processes
Dan Mikulincer

2021

We study the extent to which wide neural networks may be approximated by Gaussian processes, when initialized with random weights. It is a well-established fact that as the width of a network goes to infinity, its law converges to that of a Gaussian process. We make this quantitative by establishing explicit convergence rates for the central limit theorem in an infinite-dimensional functional space, metrized with a natural transportation distance. We identify two regimes of interest; when the activation function is polynomial, its degree determines the rate of convergence, while for non-polynomial activations, the rate is governed by the smoothness of the function.

View PDFchevron_right
On the Power-Law Hessian Spectrums in Deep Learning
Yunfeng Cai

arXiv (Cornell University), 2022

It is well-known that the Hessian of deep loss landscape matters to optimization, generalization, and even robustness of deep learning. Recent works empirically discovered that the Hessian spectrum in deep learning has a two-component structure that consists of a small number of large eigenvalues and a large number of nearly-zero eigenvalues. However, the theoretical mechanism or the mathematical behind the Hessian spectrum is still largely under-explored. To the best of our knowledge, we are the first to demonstrate that the Hessian spectrums of welltrained deep neural networks exhibit simple power-law structures. Inspired by the statistical physical theories and the spectral analysis of natural proteins, we provide a maximum-entropy theoretical interpretation for explaining why the power-law structure exist and suggest a spectral parallel between protein evolution and training of deep neural networks. By conducing extensive experiments, we further use the power-law spectral framework as a useful tool to explore multiple novel behaviors of deep learning.

View PDFchevron_right
On random matrices arising in deep neural networks: General I.I.D. case
Leonid Pastur

Random Matrices: Theory and Applications

We study the eigenvalue distribution of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the analog of the population covariance matrix is now a function of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [J. Pennington, S. Schoenholz and S. Ganguli, The emergence of spectral universality in deep networks, Proc. Mach. Learn. Res. 84 (2018) 1924–1932, arXiv:1802.09979] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability in this case has to be justified. The justification has been given in [L. Pastur, On random matrices arising in deep neural networks: Gaussian case, Pure Appl. Funct. Anal. (2020), in press, arXiv:2001.06188] for Gauss...

View PDFchevron_right
High-dimensional dynamics of generalization error in neural networks
Haim Sompolinsky

Neural Networks, 2020

We perform an average case analysis of the generalization dynamics of large neural networks trained using gradient descent. We study the practically-relevant "high-dimensional" regime where the number of free parameters in the network is on the order of or even larger than the number of examples in the dataset. Using random matrix theory and exact solutions in linear models, we derive the generalization error and training error dynamics of learning and analyze how they depend on the dimensionality of data and signal to noise ratio of the learning problem. We find that the dynamics of gradient descent learning naturally protect against overtraining and overfitting in large networks. Overtraining is worst at intermediate network sizes, when the effective number of free parameters equals the number of samples, and thus can be reduced by making a network smaller or larger. Additionally, in the high-dimensional regime, low generalization error requires starting with small initial weights. We then turn to non-linear neural networks, and show that making networks very large does not harm their generalization performance. On the contrary, it can in fact reduce overtraining, even without early stopping or regularization of any sort. We identify two novel phenomena underlying this behavior in overcomplete models: first, there is a frozen subspace of the weights in which no learning occurs under gradient descent; and second, the statistical properties of the high-dimensional regime yield better-conditioned input correlations which protect against overtraining. We demonstrate that naive application of worst-case theories such as Rademacher complexity are inaccurate in predicting the generalization performance of deep neural networks, and derive an alternative bound which incorporates the frozen subspace and conditioning effects and qualitatively matches the behavior observed in simulation.

View PDFchevron_right
A random matrix approach to neural networks
Cosme Louart

The Annals of Applied Probability

This article studies the Gram random matrix model G = 1 T Σ T Σ, Σ = σ(W X), classically found in the analysis of random feature maps and random neural networks, where X = [x1, . . . , xT ] ∈ R p×T is a (data) matrix of bounded norm, W ∈ R n×p is a matrix of independent zero-mean unit variance entries, and σ : R → R is a Lipschitz continuous (activation) function -σ(W X) being understood entrywise. By means of a key concentration of measure lemma arising from non-asymptotic random matrix arguments, we prove that, as n, p, T grow large at the same rate, the resolvent Q = (G + γIT ) -1 , for γ > 0, has a similar behavior as that met in sample covariance matrix models, involving notably the moment Φ = T n E[G], which provides in passing a deterministic equivalent for the empirical spectral measure of G. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters. * Couillet's work is supported by the ANR Project RMT4GRAPH (ANR-14-CE28-0006).

View PDFchevron_right
Appearance of Random Matrix Theory in deep learning
rangarirai mbizi

Physica A: Statistical Mechanics and its Applications, 2022

We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover agreement with Gaussian Orthogonal Ensemble statistics across several network architectures and datasets. These results shed new light on the applicability of Random Matrix Theory to modelling neural networks and suggest a role for it in the study of loss surfaces in deep learning.

View PDFchevron_right
The empirical size of trained neural networks
Kevin Chen

Cornell University - arXiv, 2016

ReLU neural networks define piecewise linear functions of their inputs. However, initializing and training a neural network is very different from fitting a linear spline. In this paper, we expand empirically upon previous theoretical work to demonstrate features of trained neural networks. Standard network initialization and training produce networks vastly simpler than a naive parameter count would suggest and can impart odd features to the trained network. However, we also show the forced simplicity is beneficial and, indeed, critical for the wide success of these networks.

View PDFchevron_right
Knots in random neural networks
Kevin Chen

ArXiv, 2018

The weights of a neural network are typically initialized at random, and one can think of the functions produced by such a network as having been generated by a prior over some function space. Studying random networks, then, is useful for a Bayesian understanding of the network evolution in early stages of training. In particular, one can investigate why neural networks with huge numbers of parameters do not immediately overfit. We analyze the properties of random scalar-input feed-forward rectified linear unit architectures, which are random linear splines. With weights and biases sampled from certain common distributions, empirical tests show that the number of knots in the spline produced by the network is equal to the number of neurons, to very close approximation. We describe our progress towards a completely analytic explanation of this phenomenon. In particular, we show that random single-layer neural networks are equivalent to integrated random walks with variable step sizes...

View PDFchevron_right
Self-regulated complexity in neural networks
Una Qiu

Natural Computing, 2005

Recordings of spontaneous activity of in vitro neuronal networks reveal various phenomena on different time scales. These include synchronized firing of neurons, bursting events of firing on both cell and network levels, hierarchies of bursting events, etc. These findings suggest that networks' natural dynamics are self-regulated to facilitate different processes on intervals in orders of magnitude ranging from fractions of seconds to hours. Observing these unique structures of recorded time-series give rise to questions regarding the diversity of the basic elements of the sequences, the information storage capacity of a network and the means of implementing calculations.

View PDFchevron_right

References (58)

  1. Robert J. Adler and Jonathan E. Taylor. Random Fields and Geometry. Springer, 2007.
  2. N. Aronszajn. Theory of Reproducing Kernels. Transactions of the American Mathematical Society, 68(3):337-404, 1950.
  3. Kendall Atkinson and Weimin Han. Spherical harmonics and approximations on the unit sphere: an introduction, volume 2044. Springer Science & Business Media, 2012.
  4. Jean-Marc Azais and Mario Wschebor. Level Sets and Extrema of Random Processes and Fields. Wiley, 2009.
  5. Shayan Aziznejad, Joaquim Campos, and Michael Unser. Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variation. SIAM Journal on Mathematics of Data Science, 5(2):422-445, 2023.
  6. Krishnakumar Balasubramanian, Larry Goldstein, Nathan Ross, and Adil Salim. Gaussian ran- dom field approximation via Stein's method with applications to wide random neural networks. Appl. Comput. Harmon. Anal., 72:Paper No. 101668, 27, 2024.
  7. Peter. Bartlett, Dylan Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
  8. Peter Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC- dimension and Pseudodimension Bounds for Piecewise Linear Neural Networks. Journal of Machine Learning Research, 20(63):1-17, 2019.
  9. Peter Bartlett, Vitaly Maiorov, and Ron Meir. Almost Linear VC Dimension Bounds for Piece- wise Polynomial Networks. Advances in Neural Information Processing Systems (NeurIPS), 11, 1998.
  10. Nils Bertschinger, Thomas Natschläger, and Robert Legenstein. At the Edge of Chaos: Real- time Computations and Self-Organized Criticality in Recurrent Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 17, 2004.
  11. Monica Bianchini and Franco Scarselli. On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures. IEEE Transactions on Neural Networks and Learning Systems, 25(8):1553-1565, 2014.
  12. Alberto Bietti. Approximation and learning with deep convolutional models: a kernel perspec- tive. International Conference on Learning Representations (ICLR), 2022.
  13. Alberto Bietti and Francis Bach. Deep Equals Shallow for ReLU Networks in Kernel Regimes. International Conference on Learning Representations (ICLR), 2021.
  14. Bastian Bohn, Michael Griebel, and Christian Rieger. A Representer Theorem for Deep Kernel Learning. Journal of Machine Learning Research, 20(64):1-32, 2019.
  15. Francesco Cagnetta, Alessandro Favero, and Matthieu Wyart. What can be learnt with wide convolutional neural networks? In International Conference on Machine Learning, pages 3347- 3379. PMLR, 2023.
  16. Valentina Cammarota, Domenico Marinucci, Michele Salvi, and Stefano Vigogna. A quantita- tive functional central limit theorem for shallow neural networks. Modern Stochastics: Theory and Applications, 11(1):85-108, 2023.
  17. Youngmin Cho and Lawrence Saul. Kernel Methods for Deep Learning. Advances in Neural Information Processing Systems (NeurIPS), 22, 2009.
  18. Youngmin Cho and Lawrence K. Saul. Analysis and Extension of Arc-Cosine Kernels for Large Margin Classification. arXiv:1112.3712, 2011.
  19. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303-314, 1989.
  20. Amit Daniely. Depth separation for neural networks. Proceedings of the 2017 Conference on Learning Theory (ICML), PMLR 65:690-696, 2017.
  21. Amit Daniely, Roy Frostig, and Yoram Singer. Toward Deeper Understanding of Neural Net- works: The Power of Initialization and a Dual View on Expressivity. Advances in Neural Infor- mation Processing Systems (NeurIPS), 29, 2016.
  22. I. Daubechies, R. DeVore, S. Foucart, B. Hanin, and G. Petrova. Nonlinear Approximation and (Deep) ReLU Networks. Constructive Approximation, 55(1):127-172, 2022.
  23. Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahra- mani. Gaussian Process Behaviour in Wide Deep Neural Networks. International Conference on Learning Representations (ICLR), 2018.
  24. Simmaco Di Lillo. PhD Thesis. in preparation.
  25. Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. 29th Annual Conference on Learning Theory (COLT), PMLR 49:907-940, 2016.
  26. Stefano Favaro, Boris Hanin, Domenico Marinucci, Ivan Nourdin, and Giovanni Peccati. Quan- titative CLTs in Deep Neural Networks. arXiv:2307.06092, 2023.
  27. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 9:249-256, 2010.
  28. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep Sparse Rectifier Neural Networks. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AIS- TATS), PMLR 15:315-323, 2011.
  29. Krzysztof M Gorski, Eric Hivon, Anthony J Banday, Benjamin D Wandelt, Frode K Hansen, Mstvos Reinecke, and Matthia Bartelmann. HEALPix: A Framework for High-Resolution Dis- cretization and Fast Analysis of Data Distributed on the Sphere. The Astrophysical Journal, 622(2):759, 2005.
  30. Alexis Goujon, Arian Etemadi, and Michael Unser. On the number of regions of piecewise linear neural networks. Journal of Computational and Applied Mathematics, 441:115667, 2024.
  31. Boris Hanin. Random neural networks in the infinite width limit as Gaussian processes. The Annals of Applied Probability, 33(6A):4798 -4819, 2023.
  32. Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. Proceedings of the 36th International Conference on Machine Learning (ICML), PMLR 97:2596-2604, 2019.
  33. Boris Hanin and David Rolnick. Deep ReLU Networks Have Surprisingly Few Activation Pat- terns. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
  34. Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the Impact of the Activation function on Deep Neural Networks Training. Proceedings of the 36th International Conference on Machine Learning (ICML), PMLR 97:2672-2680, 2019.
  35. Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251-257, 1991.
  36. Wentao Huang, Houbao Lu, and Haizhang Zhang. Hierarchical Kernels in Deep Kernel Learn- ing. Journal of Machine Learning Research, 24(391):1-30, 2023.
  37. Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
  38. Annika Lang and Christoph Schwab. Isotropic Gaussian random fields on the sphere: Regular- ity, fast simulation and stochastic partial differential equations. The Annals of Applied Proba- bility, 25(6):3047-3094, 2015.
  39. Jaehoon Lee, Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep Neural Networks as Gaussian Processes. International Conference on Learning Representations (ICLR), 2018.
  40. Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861-867, 1993.
  41. Domenico Marinucci and Giovanni Peccati. Random Fields on the Sphere: Representation, Limit Theorems and Cosmological Applications. Cambridge University Press, 2011.
  42. Hrushikesh N Mhaskar and Tomaso Poggio. Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications, 14(06):829-848, 2016.
  43. Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the Number of Linear Regions of Deep Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 27, 2014.
  44. Radford M. Neal. Bayesian Learning for Neural Networks. Springer New York, 1996.
  45. Suzanna Parkinson, Greg Ongie, Rebecca Willett, Ohad Shamir, and Nathan Srebro. Depth Separation in Norm-Bounded Infinite-Width Neural Networks. arXiv:2402.08808, 2024.
  46. Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8:143-195, 1999.
  47. Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Ex- ponential expressivity in deep neural networks through transient chaos. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
  48. Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learn- ing. The MIT Press, 2005.
  49. Itay Safran and Ohad Shamir. Depth-width tradeoffs in approximating natural functions with neural networks. Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR 70:2979-2987, 2017.
  50. Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. Ann. Statist., 48(4):1875-1897, 2020.
  51. Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Informa- tion Propagation. 5th International Conference on Learning Representations (ICLR), 2017.
  52. Shizhao Sun, Wei Chen, Liwei Wang, Xiaoguang Liu, and Tie-Yan Liu. On the Depth of Deep Neural Networks: A Theoretical View. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 1(30):2066-2072, 2016.
  53. G. Szegő. Orthogonal Polynomials. American Mathematical Society, 1975.
  54. Matus Telgarsky. Benefits of depth in neural networks. 29th Annual Conference on Learning Theory (COLT), PMLR 49:1517-1539, 2016.
  55. Luca Venturi, Samy Jelassi, Tristan Ozuch, and Joan Bruna. Depth separation beyond radial functions. Journal of Machine Learning Research, 23(122):1-56, 2022.
  56. Christopher Williams. Computing with Infinite Networks. Advances in Neural Information Processing Systems (NeurIPS), 9, 1996.
  57. Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep Kernel Learning. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 51:370-378, 2016.
  58. Lechao Xiao. Eigenspace restructuring: a principle of space and frequency in neural networks. In Conference on Learning Theory, pages 4888-4944. PMLR, 2022.

Related papers

What training reveals about neural network complexity
Marinos Poiitis

ArXiv, 2021

This work explores the hypothesis that the complexity of the function a deep neural network (NN) is learning can be deduced by how fast its weights change during training. Our analysis provides evidence for this supposition by relating the network’s distribution of Lipschitz constants (i.e., the norm of the gradient at different regions of the input space) during different training intervals with the behavior of the stochastic training procedure. We first observe that the average Lipschitz constant close to the training data affects various aspects of the parameter trajectory, with more complex networks having a longer trajectory, bigger variance, and often veering further from their initialization. We then show that NNs whose biases are trained more steadily have bounded complexity even in regions of the input space that are far from any training point. Finally, we find that steady training with Dropout implies a trainingand data-dependent generalization bound that grows poly-logar...

View PDFchevron_right
On the Effect of the Activation Function on the Distribution of Hidden Nodes in a Deep Network
Hanie Sedghi

Neural Computation

We analyze the joint probability distribution on the lengths of the vectors of hidden variables in different layers of a fully connected deep network, when the weights and biases are chosen randomly according to gaussian distributions. We show that if the activation function [Formula: see text] satisfies a minimal set of assumptions, satisfied by all activation functions that we know that are used in practice, then, as the width of the network gets large, the “length process” converges in probability to a length map that is determined as a simple function of the variances of the random weights and biases and the activation function [Formula: see text]. We also show that this convergence may fail for [Formula: see text] that violate our assumptions. We show how to use this analysis to choose the variance of weight initialization, depending on the activation function, so that hidden variables maintain a consistent scale throughout the network.

View PDFchevron_right
Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training
Jose Mourao

arXiv (Cornell University), 2023

Recent developments in applications of artificial neural networks with over n = 10 14 parameters make it extremely important to study the large n behaviour of such networks. Most works studying wide neural networks have focused on the infinite width n → +∞ limit of such networks and have shown that, at initialization, they correspond to Gaussian processes . In this work we will study their behavior for large, but finite n. Our main contributions are the following: • The computation of the corrections to Gaussianity in terms of an asymptotic series in n -1 2 . The coefficients in this expansion are determined by the statistics of parameter initialization and by the activation function. • Controlling the evolution of the outputs of finite width n networks, during training, by computing deviations from the limiting infinite width case (in which the network evolves through a linear flow). This improves previous estimates and along the way we also obtain sharper decay rates for the (finite width) NTK in terms of n, valid during the entire training procedure. As a corollary, we also prove that, with arbitrarily high probability, the training of sufficiently wide neural networks converges to a global minimum of the corresponding quadratic loss function. • Estimating how the deviations from Gaussianity evolve with training in terms of n. In particular, using a certain metric in the space of measures we find that, along training, the resulting measure is within n -1 2 (log n) 1+ of the time dependent Gaussian process corresponding to the infinite width network (which is explicitly given by precomposing the initial Gaussian process with the linear flow corresponding to training in the infinite width limit).

View PDFchevron_right
A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs
Devansh Bisla

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021

This paper focuses on understanding how the generalization error scales with the amount of the training data for deep neural networks (DNNs). Existing techniques in statistical learning theory require a computation of capacity measures, such as VC dimension, to provably bound this error. It is however unclear how to extend these measures to DNNs and therefore the existing analyses are applicable to simple neural networks, which are not used in practice, e.g., linear or shallow (at most two-layer) ones or otherwise multi-layer perceptrons. Moreover many theoretical error bounds are not empirically verifiable. In this paper we derive estimates of the generalization error that hold for deep networks and do not rely on unattainable capacity measures. The enabling technique in our approach hinges on two major assumptions: i) the network achieves zero training error, ii) the probability of making an error on a test point is proportional to the distance between this point and its nearest t...

View PDFchevron_right
Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variation
Andrew Barron

2019

For any ReLU network there is a representation in which the sum of the absolute values of the weights into each node is exactly $1$, and the input layer variables are multiplied by a value $V$ coinciding with the total variation of the path weights. Implications are given for Gaussian complexity, Rademacher complexity, statistical risk, and metric entropy, all of which are shown to be proportional to $V$. There is no dependence on the number of nodes per layer, except for the number of inputs $d$. For estimation with sub-Gaussian noise, the mean square generalization error bounds that can be obtained are of order $V \sqrt{L + \log d}/\sqrt{n}$, where $L$ is the number of layers and $n$ is the sample size.

View PDFchevron_right

Related topics

  • Computer Science
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved