Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Related Work
Computation Graph Analysis
Experimental Results
CIFAR-100 and Analysis
Stanford Cars Dataset
Method
Conclusion
References
All Topics
Computer Science
Artificial Intelligence

Feedback Networks

Profile image of William ShenWilliam Shen

2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

https://doi.org/10.1109/CVPR.2017.196
visibility

…

description

10 pages

link

1 file

Abstract

Currently, the most successful learning models in computer vision are based on learning successive representations followed by a decision layer. This is usually actualized through feedforward multilayer neural networks, e.g. ConvNets, where each layer forms one of such successive representations. However, an alternative that can achieve the same goal is a feedback based approach in which the representation is formed in an iterative manner based on a feedback received from previous iteration's output. We establish that a feedback based approach has several core advantages over feedforward: it enables making early predictions at the query time, its output naturally conforms to a hierarchical structure in the label space (e.g. a taxonomy), and it provides a new basis for Curriculum Learning. We observe that feedback develops a considerably different representation compared to feedforward counterparts, in line with the aforementioned advantages. We provide a general feedback based learning architecture, instantiated using existing RNNs, with the endpoint results on par or better than existing feedforward networks and the addition of the above advantages.

Related papers

Gated Feedback Recurrent Neural Networks
Yoshua Bengio

In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pair of layers. The recurrent signals exchanged between layers are gated adaptively based on the previous hidden states and the current input. We evaluated the proposed GF-RNN with different types of recurrent units, such as tanh, long short-term memory and gated recurrent units, on the tasks of character-level language modeling and Python program evaluation. Our empirical evaluation of different RNN units, revealed that in both tasks, the GF-RNN outperforms the conventional approaches to build deep stacked RNNs. We suggest that the improvement arises because the GF-RNN can adaptively assign different layers to different timescales and layer-to-layer interactions (including the top-down ones which are not usually present in a stacked RNN) by learning to gate these interactions.

View PDFchevron_right
TSO: Curriculum Generation using continuous optimization
Mukur Gupta

2021

The training of deep learning models poses vast challenges of including parameter tuning and ordering of training data. Significant research has been done in Curriculum learning for optimizing the sequence of training data. Recent works have focused on using complex reinforcement learning techniques to find the optimal data ordering strategy to maximize learning for a given network. In this paper, we present a simple and efficient technique based on continuous optimization. We call this new approach Training Sequence Optimization (TSO). There are three critical components in our proposed approach: (a) An encoder network maps/embeds training sequence into continuous space. (b) A predictor network uses the continuous representation of a strategy as input and predicts the accuracy for fixed network architecture. (c) A decoder further maps a continuous representation of a strategy to the ordered training dataset. The performance predictor and encoder enable us to perform gradient-based ...

View PDFchevron_right
Towards holistic scene understanding: Feedback enabled cascaded classification models
Ashutosh Saxena

2010

Abstract Scene understanding includes many related subtasks, such as scene categorization, depth estimation, object detection, etc. Each of these subtasks is often notoriously hard, and state-of-the-art classifiers already exist for many of them. These classifiers operate on the same raw image and provide correlated outputs. It is desirable to have an algorithm that can capture such correlation without requiring any changes to the inner workings of any classifier.

View PDFchevron_right
Unrolling Loopy Top-Down Semantic Feedback in Convolutional Deep Networks
Joost van de Weijer

2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014

In this paper, we propose a novel way to perform topdown semantic feedback in convolutional deep networks for efficient and accurate image parsing. We also show how to add global appearance/semantic features, which have shown to improve image parsing performance in state-ofthe-art methods, and was not present in previous convolutional approaches. The proposed method is characterised by an efficient training and a sufficiently fast testing. We use the well known SIFTflow dataset to numerically show the advantages provided by our contributions, and to compare with state-of-the-art image parsing convolutional based approaches.

View PDFchevron_right
Learning Semantic Prediction using Pretrained Deep Feedforward Networks
Sven Behnke

The ability to predict future environment states is crucial for anticipative behavior of autonomous agents. Deep learning based methods have proven to solve key perception challenges but currently mainly operate in a non-predictive fashion. We bridge this gap by proposing an approach to transform trained feed-forward networks into predictive ones via a combination of a recurrent predictive module with a teacher-student training strategy. This transformation can be conducted without the need of labeled data in a fully self-supervised fashion. Using simulated data, we demonstrate the ability of the resulting model to temporally predict a task-specific representation and additionally show the benefits of using our approach even when no corresponding feed-forward model is available.

View PDFchevron_right
Prediction as a candidate for learning deep hierarchical models of data
Rasmus Berg Palm

Recent findings [HOT06] have made possible the learning of deep layered hier- archical representations of data mimicking the brains working. It is hoped that this paradigm will unlock some of the power of the brain and lead to advances towards true AI. In this thesis I implement and evaluate state-of-the-art deep learning models and using these as building blocks I investigate the hypothesis that predicting the time-to-time sensory input is a good learning objective. I introduce the Predictive Encoder (PE) and show that a simple non-regularized learning rule, minimizing prediction error on natural video patches leads to receptive fields similar to those found in Macaque monkey visual area V1. I scale this model to video of natural scenes by introducing the Convolutional Predictive Encoder (CPE) and show similar results. Both models can be used in deep architectures as a deep learning module.

View PDFchevron_right
Towards Learning a Vocabulary of Visual Concepts and Operators using Deep Neural Networks
Sunil Kumar Vengalil

arXiv: Computer Vision and Pattern Recognition, 2021

this study, we analyze the learned feature maps of trained models using MNIST images for achieving more explainable predictions.Our study is focused on deriving a set of primitive elements, here called visual concepts, that can be used to generate any arbitrary sample from the data generating distribution.We derive the primitive elements from the feature maps learned by the model.We illustrate the idea by generating visual concepts from a Variational Autoencoder trained using MNIST images.We augment the training data of MNIST dataset by adding about 60,000 new images generated with visual concepts chosen at random.With this we were able to reduce the reconstruction loss (mean square error) from an initial value of 120 without augmentation to 60 with augmentation.Our approach is a first step towards the final goal of achieving trained deep neural network models whose predictions, features in hidden layers and the learned filters can be well explained.Such a model when deployed in production can easily be modified to adapt to new data, whereas existing deep learning models need a re training or fine tuning. This process again needs a huge number of data samples that are not easy to generate unless the model has good explainability.

View PDFchevron_right
Self-Teaching Networks
Eric Sun

Interspeech 2019, 2019

We propose self-teaching networks to improve the generalization capacity of deep neural networks. The idea is to generate soft supervision labels using the output layer for training the lower layers of the network. During the network training, we seek an auxiliary loss that drives the lower layer to mimic the behavior of the output layer. The connection between the two network layers through the auxiliary loss can help the gradient flow, which works similar to the residual networks. Furthermore, the auxiliary loss also works as a regularizer, which improves the generalization capacity of the network. We evaluated the self-teaching network with deep recurrent neural networks on speech recognition tasks, where we trained the acoustic model using 30 thousand hours of data. We tested the acoustic model using data collected from 4 scenarios. We show that the self-teaching network can achieve consistent improvements and outperform existing methods such as label smoothing and confidence penalization.

View PDFchevron_right
Task-Driven Convolutional Recurrent Models of the Visual System
James DiCarlo

2018

Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the ImageNet task. In contrast, novel cells that incorporated two structural features, bypassing and gating, were able to boost task accuracy substantially. We extended these design principles in an automated search over thousands of model architectures, which identified novel local recurrent cells and long-range feedback conne...

View PDFchevron_right
Hierarchical Class-Based Curriculum Loss
Divya Choudhary, Palash Goyal

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 2021

Classification algorithms in machine learning often assume a flat label space. However, most real world data have dependencies between the labels, which can often be captured by using a hierarchy. Utilizing this relation can help develop a model capable of satisfying the dependencies and improving model accuracy and interpretability. Further, as different levels in the hierarchy correspond to different granularities, penalizing each label equally can be detrimental to model learning. In this paper, we propose a loss function, hierarchical curriculum loss, with two properties: (i) satisfy hierarchical constraints present in the label space, and (ii) provide non-uniform weights to labels based on their levels in the hierarchy, learned implicitly by the training paradigm. We theoretically show that the proposed hierarchical class-based curriculum loss is a tight bound of 0-1 loss among all losses satisfying the hierarchical constraints. We test our loss function on real world image dat...

View PDFchevron_right

References (69)

  1. M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3686-3693. IEEE, 2014.
  2. S. J. Ashford and L. L. Cummings. Feedback as an individ- ual resource: Personal strategies of creating information. Or- ganizational behavior and human performance, 32(3):370- 398, 1983.
  3. V. Belagiannis and A. Zisserman. Recurrent human pose estimation. arXiv preprint arXiv:1605.02914, 2016.
  4. Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Cur- riculum learning. In Proceedings of the 26th annual interna- tional conference on machine learning. ACM, 2009.
  5. W. Byeon, T. M. Breuel, F. Raue, and M. Liwicki. Scene labeling with lstm recurrent neural networks. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3547-3555, 2015.
  6. C. Cao, X. Liu, Y. Yang, Y. Yu, J. Wang, Z. Wang, Y. Huang, L. Wang, C. Huang, W. Xu, et al. Look and think twice: Cap- turing top-down visual attention with feedback convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2956-2964, 2015.
  7. J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose estimation with iterative error feedback. arXiv preprint arXiv:1507.06550, 2015.
  8. R. M. Cichy, D. Pantazis, and A. Oliva. Resolving human object recognition in space and time. Nature neuroscience, 17(3):455-462, 2014.
  9. J. Deng, N. Ding, Y. Jia, A. Frome, K. Murphy, S. Bengio, Y. Li, H. Neven, and H. Adam. Large-scale object classifica- tion using label relation graphs. In European Conference on Computer Vision, pages 48-64. Springer, 2014.
  10. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.
  11. N. Ding, J. Deng, K. P. Murphy, and H. Neven. Probabilistic label relation graphs with ising models. In Proceedings of the IEEE International Conference on Computer Vision, pages 1161-1169, 2015.
  12. J. L. Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
  13. F. A. Ford. Modeling the environment: an introduction to system dynamics models of environmental systems. Island Press, 1999.
  14. C. D. Gilbert and M. Sigman. Brain states: top-down influ- ences in sensory processing. Neuron, 54(5):677-696, 2007.
  15. G. Gkioxari, A. Toshev, and N. Jaitly. Chained predic- tions using convolutional neural networks. arXiv preprint arXiv:1605.02346, 2016.
  16. K. Greff, R. K. Srivastava, and J. Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771, 2016.
  17. K. Gregor and Y. LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th International Con- ference on Machine Learning (ICML-10), pages 399-406, 2010.
  18. D. Ha, A. Dai, and Q. V. Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.
  19. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn- ing for image recognition. arXiv preprint arXiv:1512.03385, 2015.
  20. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027, 2016.
  21. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, Nov. 1997.
  22. C. S. Holling. Resilience and stability of ecological systems. Annual review of ecology and systematics, pages 1-23, 1973.
  23. H. Hu, G.-T. Zhou, Z. Deng, Z. Liao, and G. Mori. Learn- ing structured inference neural networks with label relations. arXiv preprint arXiv:1511.05616, 2015.
  24. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382, 2016.
  25. J. Hupé, A. James, B. Payne, S. Lomber, P. Girard, and J. Bullier. Cortical feedback improves discrimination be- tween figure and background by v1, v2 and v3 neurons. Na- ture, 394(6695):784-787, 1998.
  26. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
  27. A. Karpathy and L. Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128-3137, 2015.
  28. D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
  29. J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object rep- resentations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
  30. A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009.
  31. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012.
  32. K. A. Krueger and P. Dayan. Flexible shaping: How learning in small steps helps. Cognition, 110(3):380-394, 2009.
  33. S. Kumar and W. Byrne. Minimum bayes-risk decoding for statistical machine translation. Technical report, DTIC Doc- ument, 2004.
  34. E. B. Lee and L. Markus. Foundations of optimal control theory. Technical report, DTIC Document, 1967.
  35. T. S. Lee and D. Mumford. Hierarchical bayesian inference in the visual cortex. JOSA A, 20(7):1434-1448, 2003.
  36. K. Li, B. Hariharan, and J. Malik. Iterative instance segmen- tation. arXiv preprint arXiv:1511.08498, 2015.
  37. M. Liang and X. Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 3367-3375, 2015.
  38. Q. Liao and T. Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint arXiv:1604.03640, 2016.
  39. T.-Y. Lin, A. RoyChowdhury, and S. Maji. Bilinear cnn mod- els for fine-grained visual recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 1449-1457, 2015.
  40. L. v. d. Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.
  41. V. Mnih, N. Heess, A. Graves, et al. Recurrent models of vi- sual attention. In Advances in neural information processing systems, pages 2204-2212, 2014.
  42. A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. arXiv preprint arXiv:1603.06937, 2016.
  43. M. Oberweger, P. Wohlhart, and V. Lepetit. Training a feed- back loop for hand pose estimation. In Proceedings of the IEEE International Conference on Computer Vision, pages 3316-3324, 2015.
  44. A. G. Parlos, K. T. Chong, and A. F. Atiya. Application of the recurrent multilayer perceptron in modeling complex process dynamics. IEEE Transactions on Neural Networks, 5(2):255-266, 1994.
  45. P. H. Pinheiro and R. Collobert. Recurrent convolutional neural networks for scene labeling. In ICML, pages 82-90, 2014.
  46. V. Ramakrishna, D. Munoz, M. Hebert, J. A. Bagnell, and Y. Sheikh. Pose machines: Articulated pose estimation via inference machines. In European Conference on Computer Vision, pages 33-47. Springer, 2014.
  47. N. C. Rust and J. J. DiCarlo. Selectivity and tolerance (invariance) both increase as visual information propagates from cortical area v4 to it. The Journal of Neuroscience, 30(39):12978-12995, 2010.
  48. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  49. S. Singh, D. Hoiem, and D. Forsyth. Swapout: Learn- ing an ensemble of deep architectures. arXiv preprint arXiv:1605.06465, 2016.
  50. R. Socher, B. Huval, B. P. Bath, C. D. Manning, and A. Y. Ng. Convolutional-recursive deep learning for 3d object clas- sification. In NIPS, volume 3, page 8, 2012.
  51. R. Socher, C. C. Lin, C. Manning, and A. Y. Ng. Parsing nat- ural scenes and natural language with recursive neural net- works. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 129-136, 2011.
  52. Y. Song, M. Zhao, J. Yagnik, and X. Wu. Taxonomic classi- fication for web-based videos. In Computer Vision and Pat- tern Recognition (CVPR), 2010 IEEE Conference on, pages 871-878. IEEE, 2010.
  53. R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015.
  54. M. F. Stollenga, J. Masci, F. Gomez, and J. Schmidhuber. Deep networks with internal selective attention through feed- back connections. In Advances in Neural Information Pro- cessing Systems, pages 3545-3553, 2014.
  55. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.
  56. J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bre- gler. Efficient object localization using convolutional net- works. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 648-656, 2015.
  57. A. Toshev and C. Szegedy. Deeppose: Human pose estima- tion via deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1653-1660, 2014.
  58. Z. Tu. Auto-context and its application to high-level vision tasks. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1-8. IEEE, 2008.
  59. A. Veit, M. J. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems, pages 550-558, 2016.
  60. Q. Wang, J. Zhang, S. Song, and Z. Zhang. Attentional neu- ral network: Feature selection using cognitive feedback. In Advances in Neural Information Processing Systems, pages 2033-2041, 2014.
  61. Z. Wang, S. Chang, J. Zhou, M. Wang, and T. S. Huang. Learning a task-specific deep architecture for clustering. In Proceedings of the 2016 SIAM International Conference on Data Mining, pages 369-377. SIAM, 2016.
  62. D. J. Weiss and B. Taskar. Structured prediction cascades. In AISTATS, pages 916-923, 2010.
  63. D. H. Wolpert. Stacked generalization. Neural networks, 5(2):241-259, 1992.
  64. D. Wyatte, T. Curran, and R. O'Reilly. The limits of feed- forward vision: Recurrent processing promotes robust object recognition when objects are degraded. Journal of Cognitive Neuroscience, 24(11):2248-2261, 2012.
  65. S. Xie, T. Yang, X. Wang, and Y. Lin. Hyper-class aug- mented and regularized deep learning for fine-grained im- age classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2645- 2654, 2015.
  66. S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c. Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Advances in Neural Information Processing Systems, 2015.
  67. K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhut- dinov, R. S. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2(3):5, 2015.
  68. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European Conference on Com- puter Vision, pages 818-833. Springer, 2014.
  69. S. Zhang, Y. Wu, T. Che, Z. Lin, R. Memisevic, R. Salakhut- dinov, and Y. Bengio. Architectural complexity measures of recurrent neural networks. arXiv preprint arXiv:1602.08210, 2016.

Related papers

Incorporating Feedback in Convolutional Neural Networks
Heiko Neumann

2019 Conference on Cognitive Computational Neuroscience

Convolutional neural networks (CNNs) are a state-of-theart machine learning method, partially inspired by the hierarchical structure of cortex. They typically process information from input to output in a feedforward manner. It has been shown that incorporating feedback pathways can improve their performance and robustness. However, little is known about why feedback helps and how feedforward and feedback signals are best combined. Here, we compare feedforward and feedback networks using a multi-digit classification task, quantifying performance as well as robustness against image noise. We show that the advantage of feedback networks which add the feedback to the feedforward signal is largely due to the increased receptive field size of their neurons. In addition, we show that networks which use modulating or subtractive feedback (inspired by theories of feedback processing in cortex) outperform additive architectures and have increased robustness against noise. These results provide a first step towards using feedback in convolutional neural networks more effectively.

View PDFchevron_right
Spatial Transformer Networks for Curriculum Learning
Fatemeh Azimi

2021

Curriculum learning is a bio-inspired training technique that is widely adopted to machine learning for improved optimization and better training of neural networks regarding the convergence rate or obtained accuracy. The main concept in curriculum learning is to start the training with simpler tasks and gradually increase the level of difficulty. Therefore, a natural question is how to determine or generate these simpler tasks. In this work, we take inspiration from Spatial Transformer Networks (STNs) in order to form an easy-to-hard curriculum. As STNs have been proven to be capable of removing the clutter from the input images and obtaining higher accuracy in image classification tasks, we hypothesize that images processed by STNs can be seen as easier tasks and utilized in the interest of curriculum learning. To this end, we study multiple strategies developed for shaping the training curriculum, using the data generated by STNs. We perform various experiments on cluttered MNIST...

View PDFchevron_right
Recurrent Feedback Improves Feedforward Representations in Deep Neural Networks
Tai Lee

arXiv (Cornell University), 2019

The abundant recurrent horizontal and feedback connections in the primate visual cortex are thought to play an important role in bringing global and semantic contextual information to early visual areas during perceptual inference, helping to resolve local ambiguity and fill in missing details. In this study, we find that introducing feedback loops and horizontal recurrent connections to a deep convolution neural network (VGG16) allows the network to become more robust against noise and occlusion during inference, even in the initial feedforward pass. This suggests that recurrent feedback and contextual modulation transform the feedforward representations of the network in a meaningful and interesting way. We study the population codes of neurons in the network, before and after learning with feedback, and find that learning with feedback yielded an increase in discriminability (measured by d-prime) between the different object classes in the population codes of the neurons in the feedforward path, even at the earliest layer that receives feedback. We find that recurrent feedback, by injecting top-down semantic meaning to the population activities, helps the network learn better feedforward paths to robustly map noisy image patches to the latent representations corresponding to important visual concepts of each object class, resulting in greater robustness of the network against noises and occlusion as well as better fine-grained recognition.

View PDFchevron_right
Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum
Dennis DeCoste

2019

Recent works have shown that learning from easier instances first can help deep neural networks (DNNs) generalize better. However, knowing which data to present during different stages of training is a challenging problem. In this work, we address this problem by introducing data parameters. More specifically, we equip each sample and class in a dataset with a learnable parameter (data parameters), which governs their importance in the learning process. During training, at each iteration, as we update the model parameters, we also update the data parameters. These updates are done by gradient descent and do not require hand-crafted rules or design. When applied to image classification task on CIFAR10, CIFAR100,WebVision and ImageNet datasets, and object detection task on KITTI dataset, learning a dynamic curriculum via data parameters leads to consistent gains, without any increase in model complexity or training time. When applied to a noisy dataset, the proposed method learns to l...

View PDFchevron_right
Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks
gad cohen

ArXiv, 2018

Our first contribution in this paper is a theoretical investigation of curriculum learning in the context of stochastic gradient descent when optimizing the least squares loss function. We prove that the rate of convergence of an ideal curriculum learning method in monotonically increasing with the difficulty of the examples, and that this increase in convergence rate is monotonically decreasing as training proceeds. In our second contribution we analyze curriculum learning in the context of training a CNN for image classification. Here one crucial problem is the means to achieve a curriculum. We describe a method which infers the curriculum by way of transfer learning from another network, pre-trained on a different task. While this approach can only approximate the ideal curriculum, we observe empirically similar behavior to the one predicted by the theory, namely, a significant boost in convergence speed at the beginning of training. When the task is made more difficult, improvem...

View PDFchevron_right

Related topics

  • Computer Science
  • Artificial Intelligence
  • Feedforward Neural Network
  • Representation Politics

    • Cited by

    Beyond the feedforward sweep: feedback computations in the visual cortex
    Gabriel Kreiman

    Annals of the New York Academy of Sciences

    The goal of our review is to bring together recent exciting and complementary developments in computational cognitive neuroscience, with behavioral and neurophysiological results as the first step toward a unifying theory for how our visual system integrates bottom-up sensory inputs with top-down mnemonic and cognitive processes.

    View PDFchevron_right
    Massively Parallel Video Networks
    Andrew Zisserman

    Computer Vision – ECCV 2018

    We introduce a class of causal video understanding models that aims to improve efficiency of video processing by maximising throughput, minimising latency, and reducing the number of clock cycles. Leveraging operation pipelining and multi-rate clocks, these models perform a minimal amount of computation (e.g. as few as four convolutional layers) for each frame per timestep to produce an output. The models are still very deep, with dozens of such operations being performed but in a pipelined fashion that enables depth-parallel computation. We illustrate the proposed principles by applying them to existing image architectures and analyse their behaviour on two video tasks: action recognition and human keypoint localisation. The results show that a significant degree of parallelism, and implicitly speedup, can be achieved with little loss in performance.

    View PDFchevron_right
    AFN: Attentional Feedback Network Based 3D Terrain Super-Resolution
    Ashish Kubade

    Computer Vision – ACCV 2020

    Terrain, representing features of an earth surface, plays a crucial role in many applications such as simulations, route planning, analysis of surface dynamics, computer graphics-based games, entertainment, films, to name a few. With recent advancements in digital technology, these applications demand the presence of high resolution details in the terrain. In this paper, we propose a novel fully convolutional neural network based super-resolution architecture to increase the resolution of low-resolution Digital Elevation Model (LRDEM) with the help of information extracted from the corresponding aerial image as a complementary modality. We perform the super-resolution of LRDEM using an attention based feedback mechanism named 'Attentional Feedback Network' (AFN), which selectively fuses the information from LRDEM and aerial image to enhance and infuse the high-frequency features and to produce the terrain realistically . We compare the proposed architecture with existing state-of-the-art DEM super-resolution methods and show that the proposed architecture outperforms enhancing the resolution of input LRDEM accurately and in a realistic manner. 12

    View PDFchevron_right
    Distributed Iterative Gating Networks for Semantic Segmentation
    rezaul karim

    2020

    In this paper, we present a canonical structure for controlling information flow in neural networks with an efficient feedback routing mechanism based on a strategy of Distributed Iterative Gating (DIGNet). The structure of this mechanism derives from a strong conceptual foundation, and presents a light-weight mechanism for adaptive control of computation similar to recurrent convolutional neural networks by integrating feedback signals with a feed forward architecture. In contrast to other RNN formulations, DIGNet generates feedback signals in a cascaded manner that implicitly carries information from all the layers above. This cascaded feedback propagation by means of the propagator gates is found to be more effective compared to other feedback mechanisms that use feedback from output of either the corresponding stage or from the previous stage. Experiments reveal the high degree of capability that this recurrent approach with cascaded feedback presents over feed-forward baselines...

    View PDFchevron_right
    Recurrent Connections in the Primate Ventral Visual Stream Mediate a Tradeoff Between Task Performance and Network Size During Core Object Recognition
    Dan Yamins

    2021

    The computational role of the abundant feedback connections in the ventral visual stream (VVS) is unclear, enabling humans and non-human primates to effortlessly recognize objects across a multitude of viewing conditions. Prior studies have augmented feedforward convolutional neural networks (CNNs) with recurrent connections to study their role in visual processing; however, often these recurrent networks are optimized directly on neural data or the comparative metrics used are undefined for standard feedforward networks that lack these connections. In this work, we develop task-optimized convolutional recurrent (ConvRNN) network models that more correctly mimic the timing and gross neuroanatomy of the ventral pathway. Properly chosen intermediate-depth ConvRNN circuit architectures, which incorporate mechanisms of feedforward bypassing and recurrent gating, can achieve high performance on a core recognition task, comparable to that of much deeper feedforward networks. We then devel...

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved