Academia.eduAcademia.edu

Figure 15 – uploaded by Daniel Yamins

See full PDFdownloadDownload figure
Fig. $10. Tradeoffs between subtask-optimal architectures. Each panel shows pairwise relative performance of the models from the high-throughput e& periments in Fig. 1A and Fig. $3 on a variety of binary subtasks. As in that figure, random selections are shown in green and performance-optimized selectior are shown in blue. Sometimes performance on one binary subtask—e.g., Boats-vs.-Fruits and Fruits-vs.-Tables (lower right corner panel)—directly correlate with performance on another. More commonly, there is a tradeoff between subtask performance in the models explored during optimization, leading to the pattern observed in subtask pairs. Because the procedure was maximizing overall performance (as opposed to performance on any one subtask), one arm the V is heavier than the other, corresponding to the optimization process being forced to make a single choice in each of these tradeoffs.

Figure 1 $10. Tradeoffs between subtask-optimal architectures. Each panel shows pairwise relative performance of the models from the high-throughput e& periments in Fig. 1A and Fig. $3 on a variety of binary subtasks. As in that figure, random selections are shown in green and performance-optimized selectior are shown in blue. Sometimes performance on one binary subtask—e.g., Boats-vs.-Fruits and Fruits-vs.-Tables (lower right corner panel)—directly correlate with performance on another. More commonly, there is a tradeoff between subtask performance in the models explored during optimization, leading to the pattern observed in subtask pairs. Because the procedure was maximizing overall performance (as opposed to performance on any one subtask), one arm the V is heavier than the other, corresponding to the optimization process being forced to make a single choice in each of these tradeoffs.

Related Figures (15)

zation (orange dots). (B) Pursuing the correlation identified in A, a high-performing neural network was identified that matches human performance on a range of recognition tasks, the HMO model. The object categorization performance vs. IT neural predictivity correlation extends across a variety of models exhibiting a wide range of performance levels. Black circles include controls and published models; red squares are models produced during the HMO op- timization procedure. The category ideal observer (purple square) lies significantly off the main trend, but is not an actual image-computable model. The r value is computed over red and black points. For reference, light blue circles indicate performance optimized models (blue dots) from A. IT Cortex as a Neural Performance Target. Fig. 14 suggests a next step toward improved encoding models of higher ventral cortex: drive models further to the right along the x axis—if the corre- lation holds, the models will also climb on the y axis. Ideally, this would involve identifying hierarchical neural networks that per- form at or near human object recognition performance levels and validating them using rigorous tests against neural data (Fig. 24). However, the difficulty of meeting the performance challenge itself can be seen in Fig. 2B. To obtain neural refer- ence points on categorization performance, we trained linear
zation (orange dots). (B) Pursuing the correlation identified in A, a high-performing neural network was identified that matches human performance on a range of recognition tasks, the HMO model. The object categorization performance vs. IT neural predictivity correlation extends across a variety of models exhibiting a wide range of performance levels. Black circles include controls and published models; red squares are models produced during the HMO op- timization procedure. The category ideal observer (purple square) lies significantly off the main trend, but is not an actual image-computable model. The r value is computed over red and black points. For reference, light blue circles indicate performance optimized models (blue dots) from A. IT Cortex as a Neural Performance Target. Fig. 14 suggests a next step toward improved encoding models of higher ventral cortex: drive models further to the right along the x axis—if the corre- lation holds, the models will also climb on the y axis. Ideally, this would involve identifying hierarchical neural networks that per- form at or near human object recognition performance levels and validating them using rigorous tests against neural data (Fig. 24). However, the difficulty of meeting the performance challenge itself can be seen in Fig. 2B. To obtain neural refer- ence points on categorization performance, we trained linear
Fig. 2. Neural-like models via performance optimization. (A) We (7) used high-throughput computational methods to optimize the parameters of a hierarchical CNN with linear-nonlinear (LN) layers for performance on a challenging invariant object recognition task. Using new test images distinct from those used to optimize the model, we then (2) compared output of each of the model's layers to IT neural responses and the output of intermediate layers to V4 neural responses. To obtain neural data for comparison, we used chronically implanted multielectrode arrays to record the responses of mul- tiunit sites in IT and V4, obtaining the mean visually evoked response of each of 296 neural sites to ~6,000 complex images. (B) Object categorization performance results on the test images for eight-way object categorization at three increasing levels of object view variation (y axis units are 8-way cate- gorization percent-correct, chance is 12.5%). IT (green bars) and V4 (hatched green bars) neural responses, and computational models (gray and red bars) were collected on the same image set and used to train support vector ma- chine (SVM) linear classifiers from which population performance accuracy was evaluated. Error bars are computed over train/test image splits. Human subject responses on the same tasks were collected via psychophysics experi- ments (black bars); error bars are due to intersubject variation. Asa computational reference, we used the same procedure to evaluate a variety of published ventral stream models targeting several levels of the ventral hierarchy. To control for low-level confounds, we tested the (trivial) pixel model, as well as SIFT, a simple baseline computer vision model (30). We also evaluated a V1-like Gabor-based model (25), a V2-like conjunction-of- Gabors model (31), and HMAX (17, 28), a model targeted at explaining higher ventral cortex and that has receptive field sizes
Fig. 2. Neural-like models via performance optimization. (A) We (7) used high-throughput computational methods to optimize the parameters of a hierarchical CNN with linear-nonlinear (LN) layers for performance on a challenging invariant object recognition task. Using new test images distinct from those used to optimize the model, we then (2) compared output of each of the model's layers to IT neural responses and the output of intermediate layers to V4 neural responses. To obtain neural data for comparison, we used chronically implanted multielectrode arrays to record the responses of mul- tiunit sites in IT and V4, obtaining the mean visually evoked response of each of 296 neural sites to ~6,000 complex images. (B) Object categorization performance results on the test images for eight-way object categorization at three increasing levels of object view variation (y axis units are 8-way cate- gorization percent-correct, chance is 12.5%). IT (green bars) and V4 (hatched green bars) neural responses, and computational models (gray and red bars) were collected on the same image set and used to train support vector ma- chine (SVM) linear classifiers from which population performance accuracy was evaluated. Error bars are computed over train/test image splits. Human subject responses on the same tasks were collected via psychophysics experi- ments (black bars); error bars are due to intersubject variation. Asa computational reference, we used the same procedure to evaluate a variety of published ventral stream models targeting several levels of the ventral hierarchy. To control for low-level confounds, we tested the (trivial) pixel model, as well as SIFT, a simple baseline computer vision model (30). We also evaluated a V1-like Gabor-based model (25), a V2-like conjunction-of- Gabors model (31), and HMAX (17, 28), a model targeted at explaining higher ventral cortex and that has receptive field sizes
Population Representation Similarity. Characterizing the IT neural representation at the population level may be equally important for understanding object visual representation as individual IT neural sites. The representation dissimilarity matrix (RDM) is a
Population Representation Similarity. Characterizing the IT neural representation at the population level may be equally important for understanding object visual representation as individual IT neural sites. The representation dissimilarity matrix (RDM) is a
Fig. 5. V4 neural predictions. (A) Actual vs. predicted response magnitudes for a typical V4 site. V4 sites are highly visually driven, but unlike IT sites show very little categorical preference, manifesting in more abrupt changes in the image-by-image plots shown here. Red highlight indicates the best- matching model (viz., HMO layer 3). (B) Distributions of explained variances percentage for each model, over the population of all measured V4 sites (n=128). (C) Comparison of V4 neural explained variance percentage for various models. Conventions follow those used in Fig. 3.
Fig. 5. V4 neural predictions. (A) Actual vs. predicted response magnitudes for a typical V4 site. V4 sites are highly visually driven, but unlike IT sites show very little categorical preference, manifesting in more abrupt changes in the image-by-image plots shown here. Red highlight indicates the best- matching model (viz., HMO layer 3). (B) Distributions of explained variances percentage for each model, over the population of all measured V4 sites (n=128). (C) Comparison of V4 neural explained variance percentage for various models. Conventions follow those used in Fig. 3.
Fig. 4. Population-level similarity. (A) Object-level representation dissimi- larity matrices (RDMs) visualized via rank-normalized color plots (blue = Oth distance percentile, red = 100th percentile). (B) IT population and the HMO- based IT model population, for image, object, and category generalizations (SI Text). (C) Quantification of model population representation similarity to IT. Bar height indicates the spearman correlation value of a given model's RDM to the RDM for the IT neural population. The IT bar represents the Spearman-Brown corrected consistency of the IT RDM for split-halves over the IT units, establishing a noise-limited upper bound. Error bars are taken over cross-validated regression splits in the case of models and over image and unit splits in the case of neural data.
Fig. 4. Population-level similarity. (A) Object-level representation dissimi- larity matrices (RDMs) visualized via rank-normalized color plots (blue = Oth distance percentile, red = 100th percentile). (B) IT population and the HMO- based IT model population, for image, object, and category generalizations (SI Text). (C) Quantification of model population representation similarity to IT. Bar height indicates the spearman correlation value of a given model's RDM to the RDM for the IT neural population. The IT bar represents the Spearman-Brown corrected consistency of the IT RDM for split-halves over the IT units, establishing a noise-limited upper bound. Error bars are taken over cross-validated regression splits in the case of models and over image and unit splits in the case of neural data.
Fig.$1. (A) The neural representation benchmark (1) testing image set on which we collected neural data and evaluated models contained 5,760 images of 64 objects in eight categories. The image set contained three subsets, with low, medium, and high levels of object view variation. Images were placed on realistic background scenes, which were chosen randomly to be uncorrelated with object category identity. (B) The screening image set used to discover the HMO model contained 4,500 images of 36 objects in nine categories. As with any two uncorrelated samples of images from the world—such as those images seen during development vs. those seen in adult life—the overall natural statistics of the screening set images were intended to be roughly similar to those of the testing set, but the specific content was quite different. Thus, the objects, semantic categories, and background scenes used in screening were totally nonoverlapping with those used in the testing set. Moreoever, different camera, lighting and noise conditions, and a different rendering software package, were used.
Fig.$1. (A) The neural representation benchmark (1) testing image set on which we collected neural data and evaluated models contained 5,760 images of 64 objects in eight categories. The image set contained three subsets, with low, medium, and high levels of object view variation. Images were placed on realistic background scenes, which were chosen randomly to be uncorrelated with object category identity. (B) The screening image set used to discover the HMO model contained 4,500 images of 36 objects in nine categories. As with any two uncorrelated samples of images from the world—such as those images seen during development vs. those seen in adult life—the overall natural statistics of the screening set images were intended to be roughly similar to those of the testing set, but the specific content was quite different. Thus, the objects, semantic categories, and background scenes used in screening were totally nonoverlapping with those used in the testing set. Moreoever, different camera, lighting and noise conditions, and a different rendering software package, were used.
Fig. $2. In this work, we use CNN models. CNNs consist of a series of hierarchical layers, with bottom layers accepting inputs directly from image pixels, witt units form the top and intermediate layers used to support training linear classifiers for performance evaluation and linear regressors for predicting neura tuning curves. (A) Following a line of existing work, we limited the constituent operations in each layer of the hierarchy to linear-nonlinear (LN) composition: including (/) a filtering operation, implementing AND-like template matching; (i/) a simple nonlinearity, e.g., a threshold; (iii) a local pooling/aggregatior operation, such as softmax; and (iv) a local competitive normalization. These layers are combined to produce low complexity (L1), intermediate complexity (L2) and high complexity (L3) networks. All operations are repeated convolutionally at each spatial position, corresponding to the general retinotopic organizatior in the ventral stream. (B) In creating the HMO model, we allow mixture of several of these elements to model heterogenous neural populations, each actinc convolutionally on the input image. The networks are structured in a manner consistent with known features of the ventral stream, as a series of areas o roughly equal complexity, but which permit bypass projections. (C) HMO is a procedure for searching the space of CNN mixtures to maximize object recognitior performance. With several rounds of optimization, HMO creates mixtures of component modules that specialize in subtasks, without needing to prespecif what these subtasks should be. Errors from earlier rounds of optimization are analyzed and used to reweight subsequent optimization toward unsolvec portions of the problem. The complementary component modules that emerge via this process are then combined and used as input to repeat the procedures hierarchically (Materials and Methods and S/ Text).
Fig. $2. In this work, we use CNN models. CNNs consist of a series of hierarchical layers, with bottom layers accepting inputs directly from image pixels, witt units form the top and intermediate layers used to support training linear classifiers for performance evaluation and linear regressors for predicting neura tuning curves. (A) Following a line of existing work, we limited the constituent operations in each layer of the hierarchy to linear-nonlinear (LN) composition: including (/) a filtering operation, implementing AND-like template matching; (i/) a simple nonlinearity, e.g., a threshold; (iii) a local pooling/aggregatior operation, such as softmax; and (iv) a local competitive normalization. These layers are combined to produce low complexity (L1), intermediate complexity (L2) and high complexity (L3) networks. All operations are repeated convolutionally at each spatial position, corresponding to the general retinotopic organizatior in the ventral stream. (B) In creating the HMO model, we allow mixture of several of these elements to model heterogenous neural populations, each actinc convolutionally on the input image. The networks are structured in a manner consistent with known features of the ventral stream, as a series of areas o roughly equal complexity, but which permit bypass projections. (C) HMO is a procedure for searching the space of CNN mixtures to maximize object recognitior performance. With several rounds of optimization, HMO creates mixtures of component modules that specialize in subtasks, without needing to prespecif what these subtasks should be. Errors from earlier rounds of optimization are analyzed and used to reweight subsequent optimization toward unsolvec portions of the problem. The complementary component modules that emerge via this process are then combined and used as input to repeat the procedures hierarchically (Materials and Methods and S/ Text).
Fig. $3. Optimization time traces for the high-throughput experiments shown in Fig. 1. In the performance and fitting-optimized the y axis shows the op- timization criterion—in the random selection case (Left), no optimization was done, and the performance data were ignored.
Fig. $3. Optimization time traces for the high-throughput experiments shown in Fig. 1. In the performance and fitting-optimized the y axis shows the op- timization criterion—in the random selection case (Left), no optimization was done, and the performance data were ignored.
Fig. $4. Correlation of model parameters with IT predictivity for the three high-throughput experiments shown in Fig. 1. Parameters for which the correlation is significantly different from 0 are shown. Also included are several additional metrics that are not direct model parameters but that represent measurable quantities of interest for each model, e.g., model object recognition performance. The x axis is Spearman r correlation of the given parameter with IT pre- dictivity for the indicated model selection procedure, including random (Left, green bars), performance-optimized (Center, blue bars), and IT predictivity optimized (Right, red bars). Parameters are ordered by correlation value for the random condition. Performance strongly correlates with IT predictivity in all selection regimes. Number of layers (model depth) consistently correlates as well, but much more weakly. Interestingly, one obvious metric—receptive field size at the top model layer—is only very weakly associated with predictivity, because, although the best models tended to have larger receptive field sizes, a large number of poor models also shared this characteristic.
Fig. $4. Correlation of model parameters with IT predictivity for the three high-throughput experiments shown in Fig. 1. Parameters for which the correlation is significantly different from 0 are shown. Also included are several additional metrics that are not direct model parameters but that represent measurable quantities of interest for each model, e.g., model object recognition performance. The x axis is Spearman r correlation of the given parameter with IT pre- dictivity for the indicated model selection procedure, including random (Left, green bars), performance-optimized (Center, blue bars), and IT predictivity optimized (Right, red bars). Parameters are ordered by correlation value for the random condition. Performance strongly correlates with IT predictivity in all selection regimes. Number of layers (model depth) consistently correlates as well, but much more weakly. Interestingly, one obvious metric—receptive field size at the top model layer—is only very weakly associated with predictivity, because, although the best models tended to have larger receptive field sizes, a large number of poor models also shared this characteristic.
Fig. S5. (A) Optimization loss traces during the HMO procedure showing decreased loss as optimization proceeded. (B) Parameter-space trajectories during two optimization rounds shown in A (round 1 are blue dots; round 2 are red dots). This 3D plot shows parameter values for two chosen parameters (L1 filter mean and L2 pooling exponent) out of many, but it is evident that subsequent rounds of optimization (e.g., red) gravitate toward different paremeter combinations (i.e., different network architetures) than earlier rounds of optimization (e.g., blue). (C) Training performance as a function of model complexity, showing dramatic increases as components from round 1 (blue bars) and round 2 (red bars) were added. The final model (black bar) consists of 30 components identified with three complementary rounds of optimization, plus one L1 layer that, anatomically, stacks on top of those 30 components and, functionally, produces nonlinear combinations of their outputs.
Fig. S5. (A) Optimization loss traces during the HMO procedure showing decreased loss as optimization proceeded. (B) Parameter-space trajectories during two optimization rounds shown in A (round 1 are blue dots; round 2 are red dots). This 3D plot shows parameter values for two chosen parameters (L1 filter mean and L2 pooling exponent) out of many, but it is evident that subsequent rounds of optimization (e.g., red) gravitate toward different paremeter combinations (i.e., different network architetures) than earlier rounds of optimization (e.g., blue). (C) Training performance as a function of model complexity, showing dramatic increases as components from round 1 (blue bars) and round 2 (red bars) were added. The final model (black bar) consists of 30 components identified with three complementary rounds of optimization, plus one L1 layer that, anatomically, stacks on top of those 30 components and, functionally, produces nonlinear combinations of their outputs.
Fig. S6. Classification results for tasks including basic eight-way basic categorization (Top), eight-way subordinate car identification (Middle), and eight-way subordinate face identification (Bottom). Each task was assessed at low, medium, and high levels of image variation (S/ Text). Comparison was made between neural data, human data, existing models from the literature, and the HMO model outputs. The tasks span a wide range of difficulty, from low-variation basic eight-way categorization where humans perform at greater than 95% accuracy to high-variation subordinate face identification, where human performance is indistinguishable from chance.
Fig. S6. Classification results for tasks including basic eight-way basic categorization (Top), eight-way subordinate car identification (Middle), and eight-way subordinate face identification (Bottom). Each task was assessed at low, medium, and high levels of image variation (S/ Text). Comparison was made between neural data, human data, existing models from the literature, and the HMO model outputs. The tasks span a wide range of difficulty, from low-variation basic eight-way categorization where humans perform at greater than 95% accuracy to high-variation subordinate face identification, where human performance is indistinguishable from chance.
Fig. S7._ (A) Dependence of performance on number of training examples for models and neural populations. HMO model is shown in red; IT population ir solid green; V4 in dotted green; all other control models are shown in black. (B) Direct comparison of dependence of performance on number of neural sites for the IT (solid green) and V4 (dotted green) neural populations.
Fig. S7._ (A) Dependence of performance on number of training examples for models and neural populations. HMO model is shown in red; IT population ir solid green; V4 in dotted green; all other control models are shown in black. (B) Direct comparison of dependence of performance on number of neural sites for the IT (solid green) and V4 (dotted green) neural populations.
Fig. $8. Additional RDM comparisons to IT population structure. As in Fig. 3 D and E, each bar shows the Spearman correlation of an RDM for a model (or V4 population) with the RDM for the IT neural population on the same stimulus set. We show comparisons for three subsets of the test image set separated by variation level (Low, Medium, and High), as well as for the whole stimulus set (All). (A) Comparisons of RDMs at the object level, in which population rep- resentation vectors are averaged on a per-object basis before taking the pairwise correlations to make the RDM matrices. (B) More detailed image-level RDMs comparisons, with each stimulus represented separately. (C) Object-level RDMs for a variety of models and the V4 and IT neural populations.
Fig. $8. Additional RDM comparisons to IT population structure. As in Fig. 3 D and E, each bar shows the Spearman correlation of an RDM for a model (or V4 population) with the RDM for the IT neural population on the same stimulus set. We show comparisons for three subsets of the test image set separated by variation level (Low, Medium, and High), as well as for the whole stimulus set (All). (A) Comparisons of RDMs at the object level, in which population rep- resentation vectors are averaged on a per-object basis before taking the pairwise correlations to make the RDM matrices. (B) More detailed image-level RDMs comparisons, with each stimulus represented separately. (C) Object-level RDMs for a variety of models and the V4 and IT neural populations.
Fig. $9. IT explained variance for each model, fit with training/test image splits generated by (7) image generalization, a random selection process in which train and test splits contain images of the same 64 objects, but on different backgrounds and at widely different poses, positions, and sizes; (2) object gen- eralization, in which train and test images are split so that they contain no overlapping objects, so that predictions are tested for generalization across object identity as well as position, pose, size and background variation; and (3) category generalization, in which train an test images are split so that they contain nc overlapping categories, so that predictions are tested across category boundaries as well. Fig. 3E shows the corresponding results at the population RDM level.
Fig. $9. IT explained variance for each model, fit with training/test image splits generated by (7) image generalization, a random selection process in which train and test splits contain images of the same 64 objects, but on different backgrounds and at widely different poses, positions, and sizes; (2) object gen- eralization, in which train and test images are split so that they contain no overlapping objects, so that predictions are tested for generalization across object identity as well as position, pose, size and background variation; and (3) category generalization, in which train an test images are split so that they contain nc overlapping categories, so that predictions are tested across category boundaries as well. Fig. 3E shows the corresponding results at the population RDM level.
Fig. S11. Characterization of selected model parameters in terms of per-component tuning specificity vs. intercomponent diversity. Each point in this plot represents an architectural parameter in the HMO model. Parameters in the upper right corner are highly tuned but also highly diverse in their tunings between model components. See S/ Text for the definition of diversity and tuning specificity.
Fig. S11. Characterization of selected model parameters in terms of per-component tuning specificity vs. intercomponent diversity. Each point in this plot represents an architectural parameter in the HMO model. Parameters in the upper right corner are highly tuned but also highly diverse in their tunings between model components. See S/ Text for the definition of diversity and tuning specificity.
Related topics:
NeuroscienceComputer VisionComputational NeuroscienceVisual Neuroscience
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved