Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Experimental Setup
Results and Analysis
Analysis of Benchpress and Clgen Language Models
Conclusion
Related Work
References

BenchPress

Profile image of Ajitha RajanAjitha Rajan

Proceedings of the International Conference on Parallel Architectures and Compilation Techniques

https://doi.org/10.1145/3559009.3569644
visibility

…

description

12 pages

link

1 file

Abstract

Finding the right heuristics to optimize code has always been a difficult and mostly manual task for compiler engineers. Today this task is near-impossible as hardware-software complexity has scaled up exponentially. Predictive models for compilers have recently emerged which require little human effort but are far better than humans in finding near optimal heuristics. As any machine learning technique, they are only as good as the data they are trained on but there is a severe shortage of code for training compilers. Researchers have tried to remedy this with code generation but their synthetic benchmarks, although thousands, are small, repetitive and poor in features, therefore ineffective. This indicates the shortage is of feature quality more than corpus size. It is more important than ever to develop a directed program generation approach that will produce benchmarks with valuable features for training compiler heuristics. We develop BenchPress, the first ML benchmark generator for compilers that is steerable within feature space representations of source code. BenchPress synthesizes compiling functions by adding new code in any part of an empty or existing sequence by jointly observing its left and right context, achieving excellent compilation rate. BenchPress steers benchmark generation towards desired target features that has been impossible for state of the art synthesizers (or indeed humans) to reach. It performs better in targeting the features of Rodinia benchmarks in 3 different feature spaces compared with (a) CLgen-a state of the art ML

Related papers

Compiler Optimization using Various Machine Learning Techniques
SINI ANNA ALEX

2019

In this paper, we discuss the survey of various research papers to find an optimization method for compiler using machine learning techniques. Relationship between compiler optimization and machine learning are described and main concepts of features, models, training, and deployment are introduced. These days compiler have lot of optimization techniques, but choosing a correct optimization will have a great impact. Logically incorrect program is compared with each correct program that is identified based on the use of hashtags uniquely to find errors.

View PDFchevron_right
Continuous learning of compiler heuristics
Stefano crespi reghizzi

ACM Transactions on Architecture and Code Optimization, 2013

Optimizing programs to exploit the underlying hardware architecture is an important task. Much research has been done on enabling compilers to find the best set of code optimizations that can build the fastest and less resource-hungry executable for a given program. A common approach is iterative compilation, sometimes enriched by machine learning techniques. This provides good results, but requires extremely long compilation times and an initial training phase lasting even for days or weeks. We present long-term learning, a new algorithm that allows the compiler user to improve the performance of compiled programs with reduced compilation times with respect to iterative compilation, and without an initial training phase. Our algorithm does not just build good programs: it acquires knowledge every time a program is compiled and it uses such knowledge to learn compiler heuristics, without the need for an expert to manually define them. The heuristics are evolved during every compilat...

View PDFchevron_right
Automatically constructing compiler optimization heuristics using supervised learning
Eliot Moss

2005

Eliot Moss has been a great thesis advisor. He has helped me to become a better researcher by shaping my critical thinking as well as by improving my expressive skills. I would like to thank the members of my thesis committee, Andy Barto, Emery Berger, and Wayne Burleson for their feedback and advice that helped to improve the overall quality of this dissertation. I gratefully acknowledge the friendships and interactions from all members of the Architecture and Language Implementation group (ALI). Beginning with my first lab meeting talk, I have received helpful feedback on the best way to present myself and my work. The ongoing discussions in the lab helped to stimulate my research. Thanks especially to M. Tyler Maxwell for some of the amazing diagrams in this dissertation. Robbie Moll was helpful at stimulating my research interests in the applications of machine learning and for believing in me as an instructor. I especially would like to acknowledge Emmanuel Agu, who has been a good friend and with whom I have had many rewarding discussions on research and life. Finally, I am extremely grateful for the love and support of my entire family. Overall, I am extremely lucky to be part of such a close and wonderful family. I would like to express my sincerest gratitude to my mother, Maria. As a young child I remember my mother always telling me that I could accomplish anything that I set my mind to. She was right as always. Her confidence in me gave me the strength both to overcome any difficulties and to maintain high goals.

View PDFchevron_right
CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research
Jason Ansel

ArXiv, 2021

Interest in applying Artificial Intelligence (AI) techniques to compiler optimizations is increasing rapidly, but compiler research has a high entry barrier. Unlike in other domains, compiler and AI researchers do not have access to the datasets and frameworks that enable fast iteration and development of ideas, and getting started requires a significant engineering investment. What is needed is an easy, reusable experimental infrastructure for real world compiler optimization tasks that can serve as a common benchmark for comparing techniques, and as a platform to accelerate progress in the field. We introduce CompilerGym, a set of environments for real world compiler optimization tasks, and a toolkit for exposing new optimization tasks to compiler researchers. CompilerGym enables anyone to experiment on production compiler optimization problems through an easy-to-use package, regardless of their experience with compilers. We build upon the popular OpenAI Gym interface enabling res...

View PDFchevron_right
Using machines to learn method-specific compilation strategies
Ricardo Sanchez

International Symposium on Code Generation and Optimization (CGO 2011), 2011

Support Vector Machines (SVMs) are used to discover method-specific compilation strategies in Testarossa, a commercial Just-in-Time (JiT) compiler employed in the IBM R J9 Java TM Virtual Machine. The learning process explores a large number of different compilation strategies to generate the data needed for training models. The trained machine-learned model is integrated with the compiler to predict a compilation plan that balances code quality and compilation effort on a per-method basis. The machinelearned plans outperform the original Testarossa for start-up performance, but not for throughput performance, for which Testarossa has been highly hand-tuned for many years.

View PDFchevron_right
A Survey on Compiler Autotuning using Machine Learning
Amir Ashouri

ACM Computing Surveys

Since the mid-1990s, researchers have been trying to use machine-learning-based approaches to solve a number of different compiler optimization problems. These techniques primarily enhance the quality of the obtained results and, more importantly, make it feasible to tackle two main compiler optimization problems: optimization selection (choosing which optimizations to apply) and phase-ordering (choosing the order of applying optimizations). The compiler optimization space continues to grow due to the advancement of applications, increasing number of compiler optimizations, and new target architectures. Generic optimization passes in compilers cannot fully leverage newly introduced optimizations and, therefore, cannot keep up with the pace of increasing options. This survey summarizes and classifies the recent advances in using machine learning for the compiler optimization field, particularly on the two major problems of (1) selecting the best optimizations, and (2) the phase-order...

View PDFchevron_right
MILEPOST GCC: Machine Learning Based Research Compiler
Phil Barnard, Cupertino Miranda, Bilha Mendelson

2008

Tuning hardwired compiler optimizations for rapidly evolving hardware makes porting an optimizing compiler for each new platform extremely challenging. Our radical approach is to develop a modular, extensible, self-optimizing compiler that automatically learns the best optimization heuristics based on the behavior of the platform. In this paper we describe MILEPOST 1 GCC, a machine-learning-based compiler that automatically adjusts its optimization heuristics to improve the execution time, code size, or compilation time of specific programs on different architectures. Our preliminary experimental results show that it is possible to considerably reduce execution time of the MiBench benchmark suite on a range of platforms entirely automatically.

View PDFchevron_right
A Survey of Machine Intelligence Heuristics in Modern Compiler Design
Miskin Dash

2019

This survey paper aims to illustrate the evolution in techniques for optimization of compilers. Optimization of compilers is a crucial task, and is often the most time expensive criteria. With the plethora of different compilers available for the same applications, it is paramount to use the best trade off available to us. This trade off can be in terms of cost or compilation time. Compiler suites that are comprehensively optimized offer a nimiety of benefits to developers and consumers alike, producing code almost 20-30% faster than standard benchmarks. It is beneficial to the user as well, extending lower bandwidth and catering to multiple such users at a time. Loftier optimization heuristics allow cost reduction in terms of processing power required, and leverage the best out of the current hardware architectures available. To this end, it is imperative to employ modern advancements in Machine Intelligence for the same, and hence papers dealing with such developments have been di...

View PDFchevron_right
Portable compiler optimisation across embedded programs and microarchitectures using machine learning
Edwin Bonilla

Building an optimising compiler is a difficult and time consuming task which must be repeated for each generation of a microprocessor. As the underlying microarchitecture changes from one generation to the next, the compiler must be retuned to optimise specifically for that new system. It may take several releases of the compiler to effectively exploit a processor's performance potential, by which time a new generation has appeared and the process starts again.

View PDFchevron_right
The Next 700 ML-Enabled Compiler Optimizations
Rajiv Shailesh Chitale

arXiv (Cornell University), 2023

There is a growing interest in enhancing compiler optimizations with ML models, yet interactions between compilers and ML frameworks remain challenging. Some optimizations require tightly coupled models and compiler internals, raising issues with modularity, performance and framework independence. Practical deployment and transparency for the end-user are also important concerns. We propose ML-Compiler-Bridge to enable ML model development within a traditional Python framework while making end-to-end integration with an optimizing compiler possible and efficient. We evaluate it on both research and production use cases, for training and inference, over several optimization problems, multiple compilers and its versions, and gym infrastructures.

View PDFchevron_right

References (38)

  1. R. Bagrodia, R. Meyer, M. Takai, Yu-An Chen, Xiang Zeng, J. Martin, and Ha Yoon Song. 1998. Parsec: a parallel simulation environment for complex systems. Computer 31, 10 (1998), 77-85. https://doi.org/10.1109/2.722293
  2. Matej Balog, Alexander Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2016. DeepCoder: Learning to Write Programs. (11 2016).
  3. Rodinia Benchmarks. [n.d.]. http://lava.cs.virginia.edu/Rodinia/download.htm. [Online; accessed 25-Apr-2022].
  4. Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan, Jeremy W. Sheaffer, Sang- Ha Lee, and Kevin Skadron. 2009. Rodinia: A benchmark suite for heterogeneous computing. In 2009 IEEE International Symposium on Workload Characterization (IISWC). 44-54. https://doi.org/10.1109/IISWC.2009.5306797
  5. Bruce Collie, Philip Ginsbach, Jackson Woodruff, Ajitha Rajan, and Michael FP O'Boyle. 2020. M3: Semantic api migrations. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 90-102.
  6. Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. 2017. Synthesizing benchmarks for predictive modeling. In 2017 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 86-99. https://doi.org/ 10.1109/CGO.2017.7863731
  7. Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuandong Tian, and Hugh Leather. 2021. CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research. arXiv:2109.08267 [cs.PL]
  8. Anderson Faustino da Silva, Bruno Conde Kind, José Wesley de Souza Magalhães, Jerônimo Nunes Rocha, Breno Campos Ferreira Guimarães, and Fernando Magno Quinão Pereira. 2021. ANGHABENCH: A Suite with One Million Compilable C Benchmarks for Code-Size Reduction. In 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 378-390. https://doi.org/10.1109/ CGO51591.2021.9370322
  9. Sander de Bruin, Vadim Liventsev, and Milan Petković. 2021. Autoencoders as Tools for Program Synthesis. arXiv:2108.07129 [cs.AI]
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Im- ageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248-255. https://doi.org/10.1109/CVPR. 2009.5206848
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.
  12. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. RobustFill: Neural Program Learning under Noisy I/O. In Proceedings of the 34th International Conference on Machine Learning -Volume 70 (Sydney, NSW, Australia) (ICML'17). JMLR.org, 990-998.
  13. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155 [cs.CL]
  14. Grigori Fursin and Olivier Temam. 2011. Collective Optimization: A Practical Collaborative Approach. ACM Trans. Archit. Code Optim. 7, 4, Article 20 (dec 2011), 29 pages. https://doi.org/10.1145/1880043.1880047
  15. GitHub. [n.d.]. https://docs.github.com/en/rest. [Online; accessed 25-Apr-2022].
  16. Andrés Goens, Alexander Brauckmann, Sebastian Ertel, Chris Cummins, Hugh Leather, and Jeronimo Castrillon. 2019. A Case Study on Machine Learning for Synthesizing Benchmarks. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (Phoenix, AZ, USA) (MAPL 2019). Association for Computing Machinery, New York, NY, USA, 38-46. https://doi.org/10.1145/3315508.3329976
  17. Google. [n.d.]. https://cloud.google.com/bigquery. [Online; accessed 25-Apr- 2022].
  18. Dominik Grewe, Zheng Wang, and Michael F. P. O'Boyle. 2013. Portable mapping of data parallel programs to OpenCL for heterogeneous systems. In Proceedings of the 2013 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 1-10. https://doi.org/10.1109/CGO.2013.6494993
  19. Kavi Gupta, Peter Christensen, Xinyun Chen, and Dawn Song. 2020. Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis.
  20. Ameer Haj-Ali, Qijing (Jenny) Huang, John Xiang, William Moses, Krste Asanovic, John Wawrzynek, and Ion Stoica. 2020. AutoPhase: Juggling HLS Phase Orderings in Random Forests with Deep Reinforcement Learning. In Proceedings of Machine Learning and Systems, I. Dhillon, D. Papailiopoulos, and V. Sze (Eds.), Vol. 2. 70-81. https://proceedings.mlsys.org/paper/2020/file/ 4e732ced3463d06de0ca9a15b6153677-Paper.pdf
  21. Farah Hariri and August Shi. 2018. SRCIROR: A Toolset for Mutation Testing of C Source Code and LLVM Intermediate Representation. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). 860-863. https: //doi.org/10.1145/3238147.3240482
  22. John L. Henning. 2006. SPEC CPU2006 Benchmark Descriptions. SIGARCH Comput. Archit. News 34, 4 (sep 2006), 1-17. https://doi.org/10.1145/1186736. 1186737
  23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (nov 1997), 1735-1780. https://doi.org/10.1162/neco.1997.9. 8.1735
  24. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and Evaluating Contextual Embedding of Source Code. arXiv:2001.00059 [cs.SE]
  25. Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation. In CGO. San Jose, CA, USA, 75-88.
  26. Yann LeCun, Y. Bengio, and Geoffrey Hinton. 2015. Deep Learning. Nature 521 (05 2015), 436-44. https://doi.org/10.1038/nature14539
  27. Leandro T. C. Melo, Rodrigo G. Ribeiro, Breno C. F. Guimarães, and Fernando Magno Quintão Pereira. 2020. Type Inference for C: Applications to the Static Analysis of Incomplete Programs. ACM Trans. Program. Lang. Syst. 42, 3, Article 15 (nov 2020), 71 pages. https://doi.org/10.1145/3421472
  28. Maxwell Nye, Luke B. Hewitt, Joshua B. Tenenbaum, and Armando Solar-Lezama. 2019. Learning to Infer Program Sketches. In ICML.
  29. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Cur- ran Associates, Inc., 8024-8035. http://papers.neurips.cc/paper/9015-pytorch- an-imperative-style-high-performance-deep-learning-library.pdf
  30. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representa- tions. (02 2018).
  31. Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding by Generative Pre-Training.
  32. H. S. Seung, M. Opper, and H. Sompolinsky. 1992. Query by Committee. In Proceed- ings of the Fifth Annual Workshop on Computational Learning Theory (Pittsburgh, Pennsylvania, USA) (COLT '92). Association for Computing Machinery, New York, NY, USA, 287-294. https://doi.org/10.1145/130385.130417
  33. OpenCL specification. [n.d.]. https://www.khronos.org/registry/OpenCL/specs/ 3.0-unified/html/OpenCL_C.html. [Online; accessed 25-Apr-2022].
  34. John E. Stone, David Gohara, and Guochun Shi. 2010. OpenCL: A Parallel Pro- gramming Standard for Heterogeneous Computing Systems. Computing in Science Engineering 12, 3 (2010), 66-73. https://doi.org/10.1109/MCSE.2010.69
  35. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Net- works. In Proceedings of the 53rd Annual Meeting of the Association for Computa- tional Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, 1556-1566. https://doi.org/10.3115/v1/P15-1150
  36. Zheng Wang and Michael O'Boyle. 2018. Machine Learning in Compiler Opti- mization. Proc. IEEE 106, 11 (2018), 1879-1901. https://doi.org/10.1109/JPROC. 2018.2817118
  37. Vanya Yaneva, Ajitha Rajan, and Christophe Dubach. 2017. Compiler-assisted test acceleration on gpus for embedded software. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 35-45.
  38. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Un- derstanding Bugs in C Compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (San Jose, Cali- fornia, USA) (PLDI '11). Association for Computing Machinery, New York, NY, USA, 283-294. https://doi.org/10.1145/1993498.1993532

Related papers

MLGO: a Machine Learning Guided Compiler Optimizations Framework
Yundi Qian

Cornell University - arXiv, 2021

Leveraging machine-learning (ML) techniques for compiler optimizations has been widely studied and explored in academia. However, the adoption of ML in general-purpose, industry strength compilers has yet to happen. We propose MLGO 1 , a framework for integrating ML techniques systematically in an industrial compiler-LLVM. As a case study, we present the details and results of replacing the heuristics-based inlining-for-size optimization in LLVM with machine learned models. To the best of our knowledge, this work is the first full integration of ML in a complex compiler pass in a real-world setting. It is available in the main LLVM repository. We use two different ML algorithms: Policy Gradient and Evolution Strategies, to train the inlining-for-size model, and achieve up to 7% size reduction, when compared to state of the art LLVM-Oz. The same model, trained on one corpus, generalizes well to a diversity of real-world targets, as well as to the same set of targets after months of active development. This property of the trained models is beneficial to deploy ML techniques in real-world settings.

View PDFchevron_right
Efficient Program Compilation Through Machine Learning Techniques
Gennady Pekhimenko

Software Automatic Tuning, 2010

The wealth of available compiler optimizations leads to the dual problems of finding the best set of optimizations and the best heuristic parameters to tune each optimization. We describe how machine learning techniques, such as logistic regression, can be used to address these problems. We focus on decreasing the compile time for a static commercial compiler, while preserving the execution time. We show that we can speed up the compile process by at least a factor of two with almost the same generated code quality on the SPEC2000 benchmark suite, and that our logistic classifier achieves the same prediction quality for non-SPEC benchmarks.

View PDFchevron_right
Meta optimization: improving compiler heuristics with machine learning
Mark Stephenson

2003

Compiler writers have crafted many heuristics over the years to approximately solve NP-hard problems efficiently. Finding a heuristic that performs well on a broad range of applications is a tedious and difficult process. This paper introduces Meta Optimization, a methodology for automatically fine-tuning compiler heuristics. Meta Optimization uses machine-learning techniques to automatically search the space of compiler heuristics. Our techniques reduce compiler design complexity by relieving compiler writers of the tedium of heuristic tuning. Our machine-learning system uses an evolutionary algorithm to automatically find effective compiler heuristics. We present promising experimental results. In one mode of operation Meta Optimization creates application-specific heuristics which often result in impressive speedups. For hyperblock formation, one optimization we present in this paper, we obtain an average speedup of 23% (up to 73%) for the applications in our suite. Furthermore, by evolving a compiler's heuristic over several benchmarks, we can create effective, general-purpose heuristics. The best general-purpose heuristic our system found for hyperblock formation improved performance by an average of 25% on our training set, and 9% on a completely unrelated test set. We demonstrate the efficacy of our techniques on three different optimizations in this paper: hyperblock formation, register allocation, and data prefetching.

View PDFchevron_right
Automatic feature generation for machine learning--based optimising compilation
Edwin Bastidas Bonilla

ACM Transactions on Architecture and Code Optimization, 2014

Recent work has shown that machine learning can automate and in some cases outperform handcrafted compiler optimisations. Central to such an approach is that machine learning techniques typically rely upon summaries or features of the program. The quality of these features is critical to the accuracy of the resulting machine learned algorithm; no machine learning method will work well with poorly chosen features. However, due to the size and complexity of programs, theoretically there are an infinite number of potential features to choose from. The compiler writer now has to expend effort in choosing the best features from this space. This article develops a novel mechanism to automatically find those features that most improve the quality of the machine learned heuristic. The feature space is described by a grammar and is then searched with genetic programming and predictive modelling. We apply this technique to loop unrolling in GCC 4.3.1 and evaluate our approach on a Pentium 6. ...

View PDFchevron_right
Compiler Optimization using Machine Learning Techniques
SINI ANNA ALEX

2019

Compilation optimization is very essential in order to increase the running speed of programs as well as minimize the object file size. Machine learning algorithms are used to select the best compiler options that result in such improvements. Compiler auto-tuning refers to the process of optimizing the performance of the code during the intermediate code-generation phase of compilation. This paper deals with machine learning based compilation optimization on feature processing, compiler autotuning and compiler optimization techniques such as loop nest optimizations and automatic generation of optimization heuristics for a target processor by machine learning. It then explores the concept of evolving iterative compilers which attempts a large number of optimization strategies and choosing the best one. It proposes an approach of selecting compiler transformations – namely probabilistic optimization. Using this approach, we can achieve significant performance improvements.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved