Academia.eduAcademia.edu

Figure 7 – uploaded by Vithursan Thangarasa

See full PDFdownloadDownload figure
Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency Figure 7: Measured speedup versus theoretical speedup at varying sparsity levels for a GPT-3 layer 12k x 12k matrix multiplication (MatMul) (Lie, 2021).

Figure 7 Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency Figure 7: Measured speedup versus theoretical speedup at varying sparsity levels for a GPT-3 layer 12k x 12k matrix multiplication (MatMul) (Lie, 2021).

Related Figures (21)

Figure 1: Top-1 Accuracy vs. Training FLOPs for variants of ResNet on ImageNet. Sparse-IFT provides significant accuracy gains across different models and sparsity levels while using the same FLOP budget as its dense counterpart. Vithursan Thangarasa*! Shreyas Saxena’' Abhay Gupta! Sean Lie!
Figure 1: Top-1 Accuracy vs. Training FLOPs for variants of ResNet on ImageNet. Sparse-IFT provides significant accuracy gains across different models and sparsity levels while using the same FLOP budget as its dense counterpart. Vithursan Thangarasa*! Shreyas Saxena’' Abhay Gupta! Sean Lie!
Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency Figure 2: Different members of the Sparse-IFT family, each parameterized by a single hyperparameter (i.e., sparsity level. s). Black and white squares denote non-active and active weights, respectively. Green block indicates a non-linear activation function (e.g., ReLU). Derived with sparsity set at 50% as an example, all transformations are Iso-FLOP to the dense feedforward function fg,, making them suitable drop-in replacements for f9,. Details about each member are in Section 2.3.
Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency Figure 2: Different members of the Sparse-IFT family, each parameterized by a single hyperparameter (i.e., sparsity level. s). Black and white squares denote non-active and active weights, respectively. Green block indicates a non-linear activation function (e.g., ReLU). Derived with sparsity set at 50% as an example, all transformations are Iso-FLOP to the dense feedforward function fg,, making them suitable drop-in replacements for f9,. Details about each member are in Section 2.3.
Table 1: Cardinality of search space for sparsity masks of different members of the Sparse-IFT family.
Table 1: Cardinality of search space for sparsity masks of different members of the Sparse-IFT family.
Table 2: Sparse Wide IFT with ResNet-18 trained using var- ious sparse training methods on CIFAR-100 across different sparsity levels (columns). Best accuracy for each sparse training method is highlighted in bold. parallel branches, and hidden dimension size, respectively. Sparse Doped maintains a constant search space by allocat- ing FLOPs between a low-rank and an unstructured sparse weight matrix. Therefore, dynamic sparse training (DST) becomes crucial for effectively traversing this larger sparse parameter subspace, as discussed in Section 3.
Table 2: Sparse Wide IFT with ResNet-18 trained using var- ious sparse training methods on CIFAR-100 across different sparsity levels (columns). Best accuracy for each sparse training method is highlighted in bold. parallel branches, and hidden dimension size, respectively. Sparse Doped maintains a constant search space by allocat- ing FLOPs between a low-rank and an unstructured sparse weight matrix. Therefore, dynamic sparse training (DST) becomes crucial for effectively traversing this larger sparse parameter subspace, as discussed in Section 3.
Figure 3: The relationship between the structure and weights of Sparse-IFT ResNet-18 networks are analyzed through a graph perspective in terms of performance. Top row: we assess the relationship between Ar;,,ay and Ajm59- Bottom row: investigates the correlation between Ar and \. The Pareto curvature heatmap visually represents the classifica- tion performance, with varying color gradients symbolizing the spectrum from low to high test accuracy on CIFAR-100.
Figure 3: The relationship between the structure and weights of Sparse-IFT ResNet-18 networks are analyzed through a graph perspective in terms of performance. Top row: we assess the relationship between Ar;,,ay and Ajm59- Bottom row: investigates the correlation between Ar and \. The Pareto curvature heatmap visually represents the classifica- tion performance, with varying color gradients symbolizing the spectrum from low to high test accuracy on CIFAR-100.
Figure 4: Ablation studies with Sparse-IFT on the ResNet-18 model for CIFAR-100 across sparsity € {50%, 75%, 90%}. (left) Sparse Wide IFT trained with dynamic unstructured and structured sparsity. (middle) Sparse-IFT family members trained with RigL, where Sparse Wide performs the best. (right) Sparse Wide IFT trained in a sparse and dense manner. Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
Figure 4: Ablation studies with Sparse-IFT on the ResNet-18 model for CIFAR-100 across sparsity € {50%, 75%, 90%}. (left) Sparse Wide IFT trained with dynamic unstructured and structured sparsity. (middle) Sparse-IFT family members trained with RigL, where Sparse Wide performs the best. (right) Sparse Wide IFT trained in a sparse and dense manner. Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
Table 4: Sparse-IFT on ImageNet. Best result for each transformation and architecture is highlighted in bold.
Table 4: Sparse-IFT on ImageNet. Best result for each transformation and architecture is highlighted in bold.
Table 3: Sparse Wide IFT with various efficient architectures on CIFAR-100 across different levels of sparsity (columns).
Table 3: Sparse Wide IFT with various efficient architectures on CIFAR-100 across different levels of sparsity (columns).
Table 5: Sparse Wide IFT variants of ResNet-18 as back- bones for: (a) object detection on MS COCO, (b) semantic segmentation on Cityscapes.
Table 5: Sparse Wide IFT variants of ResNet-18 as back- bones for: (a) object detection on MS COCO, (b) semantic segmentation on Cityscapes.
Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency Figure 5: Benchmarking unstructured sparsity during (left) inference on Neural Magic’s DeepSparse runtime and (right) training acceleration on the Cerebras CS-2. In both setups, we measure the relative increase in latency or training speed for Sparse-IFT variants against the dense model.
Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency Figure 5: Benchmarking unstructured sparsity during (left) inference on Neural Magic’s DeepSparse runtime and (right) training acceleration on the Cerebras CS-2. In both setups, we measure the relative increase in latency or training speed for Sparse-IFT variants against the dense model.
Figure 6: Illustrates the dynamic interplay between the (top row) Iterative Mean Difference Bound, Arq) and test accuracy, as well as the correlation between the Ramanujan Gap, Ar and test accuracy throughout the training process. This illustrates the evolving relationship between spectral graph properties and network performance, shedding light on the connectivity dynamics of the Sparse-IFT networks trained with DST. Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
Figure 6: Illustrates the dynamic interplay between the (top row) Iterative Mean Difference Bound, Arq) and test accuracy, as well as the correlation between the Ramanujan Gap, Ar and test accuracy throughout the training process. This illustrates the evolving relationship between spectral graph properties and network performance, shedding light on the connectivity dynamics of the Sparse-IFT networks trained with DST. Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
Table 7: Evaluation on the importance of utilizing the non-linear activation across different members of Sparse-IFT with ResNet-18 on CIFAR100 across different values of sparsity (columns). Non-linear activations enhance the representational capacity of Sparse-IFT, leading to higher accuracy. All reported results are the average over 3 random seeds. C.3. Computer Vision
Table 7: Evaluation on the importance of utilizing the non-linear activation across different members of Sparse-IFT with ResNet-18 on CIFAR100 across different values of sparsity (columns). Non-linear activations enhance the representational capacity of Sparse-IFT, leading to higher accuracy. All reported results are the average over 3 random seeds. C.3. Computer Vision
Table 8: Results with ResNet-18 on CIFAR-100 across different values of sparsity (columns). Best accuracy for each sparse training method is highlighted in bold. The original dense ResNet-18 model obtains an accuracy of 77.00.2. All reported results are over 3 random seeds.
Table 8: Results with ResNet-18 on CIFAR-100 across different values of sparsity (columns). Best accuracy for each sparse training method is highlighted in bold. The original dense ResNet-18 model obtains an accuracy of 77.00.2. All reported results are over 3 random seeds.
Table 10: Comparison of structured sparse and unstructured sparse methods on CIFAR-100 test accuracy on ResNet-18. Table 9: Evaluation of Sparse Wide and Sparse Parallel IFT with various compute efficient architectures on CIFAR-100 across different values of sparsity (columns). Using Sparse Parallel IFT, all architectures outperform the dense baseline by a significant margin.
Table 10: Comparison of structured sparse and unstructured sparse methods on CIFAR-100 test accuracy on ResNet-18. Table 9: Evaluation of Sparse Wide and Sparse Parallel IFT with various compute efficient architectures on CIFAR-100 across different values of sparsity (columns). Using Sparse Parallel IFT, all architectures outperform the dense baseline by a significant margin.
We also set up and experimented using the method proposed by Jiang et al. (2022) to train with fine-grained sparse block structures dynamically. However, the algorithm uses agglomerative clustering which led to a much slower runtime and quickly ran out of memory even at 50% sparsity using the Sparse Wide IFT on a single Nvidia V100 (16 GB).
We also set up and experimented using the method proposed by Jiang et al. (2022) to train with fine-grained sparse block structures dynamically. However, the algorithm uses agglomerative clustering which led to a much slower runtime and quickly ran out of memory even at 50% sparsity using the Sparse Wide IFT on a single Nvidia V100 (16 GB).
Table 11: Object detection results on COCO minival in the RetinaNet framework. Sparse Wide IFT configurations of RetinaNet outperform the dense baseline by a large margin on all metrics while using similar FLOPs.
Table 11: Object detection results on COCO minival in the RetinaNet framework. Sparse Wide IFT configurations of RetinaNet outperform the dense baseline by a large margin on all metrics while using similar FLOPs.
Table 12: Semantic segmentation results on the Cityscapes val set using DeepLabV3+. Sparse Wide IFT configurations ResNet-18 backbones outperform the dense baseline on all metrics while using similar FLOPs. D. Natural Language Processing: Experimental Settings
Table 12: Semantic segmentation results on the Cityscapes val set using DeepLabV3+. Sparse Wide IFT configurations ResNet-18 backbones outperform the dense baseline on all metrics while using similar FLOPs. D. Natural Language Processing: Experimental Settings
Table 13: Size, architecture, and learning hyperparameters (batch size and learning rate) of the GPT-3 Small model, which is trained using Chinchilla optimal configurations (+ 20 tokens per parameter)
Table 13: Size, architecture, and learning hyperparameters (batch size and learning rate) of the GPT-3 Small model, which is trained using Chinchilla optimal configurations (+ 20 tokens per parameter)
Table 14: Sizes and architecture definitions of the dense GPT-3 Small model and its Sparse Wide IFT variants Table 15: Performance Evaluation of Dense and Sparse Wide IFT GPT-3 Small Models at 50% and 75% sparsity levels across five tasks (i.e., ARC, HellaSwag, TruthfulQA, MMLU, and Winogrande) on the Open LLM Leaderboard
Table 14: Sizes and architecture definitions of the dense GPT-3 Small model and its Sparse Wide IFT variants Table 15: Performance Evaluation of Dense and Sparse Wide IFT GPT-3 Small Models at 50% and 75% sparsity levels across five tasks (i.e., ARC, HellaSwag, TruthfulQA, MMLU, and Winogrande) on the Open LLM Leaderboard
Related topics:
Computer ScienceHyperparameter Optimization
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved