Primary visual cortex straightens natural video trajectories
2021, Nature Communications
https://doi.org/10.1038/S41467-021-25939-ZAbstract
Many sensory-driven behaviors rely on predictions about future states of the environment. Visual input typically evolves along complex temporal trajectories that are difficult to extrapolate. We test the hypothesis that spatial processing mechanisms in the early visual system facilitate prediction by constructing neural representations that follow straighter temporal trajectories. We recorded V1 population activity in anesthetized macaques while presenting static frames taken from brief video clips, and developed a procedure to measure the curvature of the associated neural population trajectory. We found that V1 populations straighten naturally occurring image sequences, but entangle artificial sequences that contain unnatural temporal transformations. We show that these effects arise in part from computational mechanisms that underlie the stimulus selectivity of V1 cells. Together, our findings reveal that the early visual system uses a set of specialized computations to build representations that can support prediction in the natural environment.
References (61)
- Földiák, P. Learning invariance from transformation sequences. Neural Comput. 3, 194-200 (1991).
- Tishby, N., Pereira, F. C. & Bialek, W. The information bottleneck method. In Proc. 37th Annual Allerton Conference on Communication, Control and Computing (University of Illinois, Urbana, IL), Vol 37, 368-377, pages 1-16 (1999).
- Li, N. & DiCarlo, J. J. Unsupervised natural visual experience rapidly reshapes size-invariant object representation in inferior temporal cortex. Neuron 67, 1062-1075 (2010).
- Goroshin, R., Mathieu, M. & LeCun, Y. Learning to Linearize Under Uncertainty (NIPS, 2015).
- Palmer, S. E., Marre, O., Berry, M. J. & Bialek, W. Predictive information in a sensory population. Proc. Natl Acad. Sci. USA 112, 6908-6913 (2015).
- Hénaff, O. J., Goris, R. L. T. & Simoncelli, E. P. Perceptual straightening of natural videos. Nat. Neurosci. 22, 984-991 (2019).
- Tolhurst, D. J., Movshon, J. A. & Dean, A. F. The statistical reliability of signals in single neurons in cat and monkey visual cortex. Vision Res. 23, 775-785 (1983).
- Goris, R. L. T., Movshon, J. A. & Simoncelli, E. P. Partitioning neuronal variability. Nat. Neurosci. 17, 858-865 (2014).
- Cohen, M. R. & Kohn, A. Measuring and interpreting neuronal correlations. Nat. Neurosci. 14, 811 (2011).
- Moreno-Bote, R. et al. Information-limiting correlations. Nat. Neurosci. 17, 1410 (2014).
- Elsayed, G. F. & Cunningham, J. P. Structure in neural population recordings: an expected byproduct of simpler phenomena? Nat. Neurosci. 20, 1310 (2017).
- Rabinowitz, N. C., Goris, R. L., Cohen, M. & Simoncelli, E. P. Attention stabilizes the shared gain of V4 populations. eLife 4, e08998 (2015).
- Goris, R. L. T., Ziemba, C. M., Movshon, J. A. & Simoncelli, E. P. Slow gain fluctuations limit benefits of temporal integration in visual cortex. J. Vision 18, 8-8 (2018).
- Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J. & Ganguli, S. In Advances in Neural Information Processing Systems Vol. 29 (Curran Associates, Inc., 2016).
- Field, D. J. Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A 4, 2379-2394 (1987).
- Ruderman, D. L. & Bialek, W. In Advances in Neural Information Processing Systems 551-558 (1994).
- Dong, D. W. & Atick, J. J. Statistics of natural time-varying images. Network: Comput. Neural Syst. 6, 345-358 (1995).
- De Valois, R. L., Albrecht, D. G. & Thorell, L. G. Spatial frequency selectivity of cells in macaque visual cortex. Vision Res. 22, 545-559 (1982).
- Touryan, J., Felsen, G. & Dan, Y. Spatial structure of complex cell receptive fields measured with natural images. Neuron 45, 781-791 (2005).
- Hubel, D. H. & Wiesel, T. N. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. 160, 106-154.2 (1962).
- Movshon, J. A., Thompson, I. D. & Tolhurst, D. J. Receptive field organization of complex cells in the cat's striate cortex. J. Physiol. 283, 79-99 (1978).
- Adelson, E. H. & Bergen, J. R. Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A 2, 284 (1985).
- Heeger, D. J. Normalization of cell responses in cat striate cortex. Visual Neurosci. 9, 181-197 (1992).
- Carandini, M., Heeger, D. J. & Movshon, J. A. Linearity and normalization in simple cells of the macaque primary visual cortex. J. Neurosci. 17, 8621-8644 (1997).
- Touryan, J., Lau, B. & Dan, Y. Isolation of relevant visual features from random stimuli for cortical complex cells. J. Neurosci. 22, 10811-10818 (2002).
- Sharpee, T., Rust, N. C. & Bialek, W. Analyzing neural responses to natural signals: maximally informative dimensions. Neural Comput. 16, 223-250 (2004).
- Rust, N. C., Schwartz, O., Movshon, J. A. & Simoncelli, E. P. Spatiotemporal elements of macaque V1 receptive fields. Neuron 46, 945-956 (2005).
- Vintch, B., Movshon, J. A. & Simoncelli, E. P. A convolutional subunit model for neuronal responses in macaque v1. J. Neurosci. 35, 14829-14841 (2015).
- Goris, R. L. T., Simoncelli, E. P. & Movshon, J. A. Origin and function of tuning diversity in macaque visual cortex. Neuron 88, 819-831 (2015).
- Skottun, B. C. et al. Classifying simple and complex cells on the basis of response modulation. Vision Res. 31, 1078-1086 (1991).
- Fukushima, K. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybernetics 36, 193-202 (1980).
- Riesenhuber, M. & Poggio, T. Hierarchical models of object recognition in cortex. Nat. Neurosci. 2, 1019-1025 (1999).
- Wiskott, L. & Sejnowski, T. J. Slow feature analysis: unsupervised learning of invariances. Neural Comput. 14, 715-770 (2002).
- Bruna, J. & Mallat, S. Invariant scattering convolution networks. IEEE Trans. Pattern Analysis Mach. Intell. 35, 1872-1886 (2013).
- Russo, A. A. et al. Motor cortex embeds muscle-like commands in an untangled population response. Neuron 97, 953-966 (2018).
- Russo, A. A. et al. Neural trajectories in the supplementary motor area and motor cortex exhibit distinct geometries, compatible with different classes of computation. Neuron 107, 745-758 (2020).
- Sohn, H., Narain, D., Meirhaeghe, N. & Jazayeri, M. Bayesian computation through cortical latent dynamics. Neuron 103, 934-947 (2019).
- Carandini, M. et al. Do we know what the early visual system does? J. Neurosci. 25, 10577-10597 (2005).
- Cavanaugh, J. R., Bair, W. & Movshon, J. A. Nature and interaction of signals from the receptive field center and surround in macaque v1 neurons. J. Neurophysiol. 88, 2530-2546 (2002).
- Schwartz, O. & Simoncelli, E. P. Natural signal statistics and sensory gain control. Nat. Neurosci. 4, 819-825 (2001).
- Coen-Cagli, R., Kohn, A. & Schwartz, O. Flexible gating of contextual influences in natural vision. Nat. Neurosci. 18, 1648 (2015).
- Mante, V., Bonin, V. & Carandini, M. Functional mechanisms shaping lateral geniculate responses to artificial and natural stimuli. Neuron 58, 625-638 (2008).
- Berardino, A., Ballé, J., Laparra, V. & Simoncelli, E. P. Eigen-distortions of hierarchical representations. Advances in Neural Information Processing Systems 30 (eds. Guyon, I. et al.) (Curran Associates, Inc., 2017).
- Goris, R. L. T., Bai, Y. H., Henaff, O. J. & Ziemba, C. M. Perceptual straightening of natural videos arises from a cascaded computation. CoSyNe Abstracts, (III-27) (2020).
- Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl Acad. Sci. USA 111, 8619-8624 (2014).
- Khaligh-Razavi, S. M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).
- Zhuang, C. et al. Unsupervised neural network models of the ventral visual stream. Proc. Natl Acad. Sci. 118, e2014196118 (2021).
- Szegedy, C. et al. Intriguing properties of neural networks. In Proc. 2nd International Conference on Learning Representations (ICLR) (2013).
- Hénaff, O. J. & Simoncelli, E. P. Geodesics of learned representations. In Proc. 4th International Conference on Learning Representations (ICLR) (2015).
- Feather, J., Durango, A., Gonzalez, R. & McDermott, J. In Advances in Neural Information Processing Systems (eds. Wallach, H. et al.) 10078-10089 (Curran Associates, Inc., 2019).
- Singer, Y. et al. Sensory cortex is optimized for prediction of future input. Elife 7, e31557 (2018).
- Brainard, D. H. The psychophysics toolbox. Spatial Vision 10, 433-436 (1997).
- Pelli, D. G. The videotoolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision 10, 437-442 (1997).
- Nauhaus, I., Nielsen, K. J. & Callaway, E. M. Efficient receptive field tiling in primate v1. Neuron 91, 893-904 (2016).
- Seshadrinathan, K., Soundararajan, R., Bovik, A. C. & Cormack, L. K. A Subjective study to evaluate video quality assessment algorithms. In SPIE Proceedings Human Vision and Electronic Imaging (2010).
- Seshadrinathan, K., Soundararajan, R., Bovik, A. C. & Cormack, L. K. Study of subjective and objective quality assessment of video. IEEE Trans. Image Process. 19, 1427-1441 (2010).
- Pachitariu, M., Steinmetz, N. A., Kadir, S. N., Carandini, M. & Harris, K. D. In Advances in Neural Information Processing Systems (eds. Lee, D. D. et al.) 4448-4456 (Curran Associates, Inc., 2016).
- Smith, M. A., Majaj, N. J. & Movshon, J. A. Dynamics of motion signaling by neurons in macaque area mt. Nat. Neurosci. 8, 220-228 (2005).
- Jordan, M. I., Ghahramani, Z., Jaakkola, T. S. & Saul, L. K. Introduction to variational methods for graphical models. Mach. Learning 37, 183-233 (1999).
- Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. In Proc. 2nd International Conference on Learning Representations (ICLR) (2013).
- Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. 3rd International Conference on Learning Representations (ICLR) (2014).