Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Related Work
Methodology
Results
Conclusion and Future Work
References
All Topics
Computer Science
Distributed Systems

Evaluating GPU Programming Models for the LUMI Supercomputer

Profile image of Michael BussmannMichael Bussmann

Supercomputing Frontiers

https://doi.org/10.1007/978-3-031-10419-0_6
visibility

…

description

23 pages

link

1 file

Abstract

It is common in the HPC community that the achieved performance with just CPUs is limited for many computational cases. The EuroHPC pre-exascale and the coming exascale systems are mainly focused on accelerators, and some of the largest upcoming supercomputers such as LUMI and Frontier will be powered by AMD Instinct™ accelerators. However, these new systems create many challenges for developers who are not familiar with the new ecosystem or with the required programming models that can be used to program for heterogeneous architectures. In this paper, we present some of the more well-known programming models to program for current and future GPU systems. We then measure the performance of each approach using a benchmark and a mini-app, test with various compilers, and tune the codes where necessary. Finally, we compare the performance, where possible, between the NVIDIA Volta (V100), Ampere (A100) GPUs, and the AMD MI100 GPU.

Related papers

OpenMPC: extended OpenMP for efficient programming and tuning on GPUs
Seyong Lee

International Journal of Computational Science and Engineering, 2013

General-purpose graphics processing units (GPGPUs) provide inexpensive, high performance platforms for compute-intensive applications. However, their programming complexity poses a significant challenge to developers. Even though the compute unified device architecture (CUDA) programming model offers better abstraction, developing efficient GPGPU code is still complex and error-prone. This paper proposes a directive-based, high-level programming model, called OpenMPC, which addresses both programmability and tunability issues on GPGPUs. We have developed a fully automatic compilation and user-assisted tuning system supporting OpenMPC. In addition to a range of compiler transformations and optimisations, the system includes tuning capabilities for generating, pruning, and navigating the search space of compilation variants. Evaluation using 14 applications shows that our system achieves 75% of the performance of the hand-coded CUDA programmes (92% if excluding one exceptional case).

View PDFchevron_right
A unified optimizing compiler framework for different GPGPU architectures
michael MANTOR

ACM Transactions on Architecture and Code Optimization, 2012

This article presents a novel optimizing compiler for general purpose computation on graphics processing units (GPGPU). It addresses two major challenges of developing high performance GPGPU programs: effective utilization of GPU memory hierarchy and judicious management of parallelism. The input to our compiler is a naïve GPU kernel function, which is functionally correct but without any consideration for performance optimization.

View PDFchevron_right
CUDACL: A tool for CUDA and OpenCL programmers
Jeff Gray

2010

Abstract Graphical Processing Unit (GPU) programming languages are used extensively for general-purpose computations. However, GPU programming languages are at a level of abstraction suitable only for use by expert parallel programmers. This paper presents a new approach through whichC'or Java programmers can access these languages without having to focus on the technical or language-specific details.

View PDFchevron_right
AEminiumGPU: An Intelligent Framework for GPU Programming
Bruno Cabral

As a consequence of the immense computational power avail-able in GPUs, the usage of these platforms for running data-intensive general purpose programs has been increasing. Since memory and pro-cessor architectures of CPUs and GPUs are substantially different, pro-grams designed for each platform are also very different and often resort to a very distinct set of algorithms and data structures. Selecting be-tween the CPU or GPU for a given program is not easy as there are variations in the hardware of the GPU, in the amount of data, and in several other performance factors. AEminiumGPU is a new data-parallel framework for developing and run-ning parallel programs on CPUs and GPUs. AEminiumGPU programs are written in a Java using Map-Reduce primitives and are compiled into hybrid executables which can run in either platforms. Thus, the de-cision of which platform is going to be used for executing a program is delayed until run-time and automatically performed by the system using Mach...

View PDFchevron_right
Performance Assessment of OpenMP Compilers Targeting NVIDIA V100 GPUs
Swaroop Pophale

Accelerator Programming Using Directives, 2021

Heterogeneous systems are becoming increasingly prevalent. In order to exploit the rich compute resources of such systems, robust programming models are needed for application developers to seamlessly migrate legacy code from today's systems to tomorrow's. Over the past decade and more, directives have been established as one of the promising paths to tackle programmatic challenges on emerging systems. This work focuses on applying and demonstrating OpenMP offloading directives on five proxy applications. We observe that the performance varies widely from one compiler to the other; a crucial aspect of our work is reporting best practices to application developers who use OpenMP offloading compilers. While some issues can be worked around by the developer, there are other issues that must be reported to the compiler vendors. By restructuring OpenMP offloading directives, we gain an 18x speedup for the su3 proxy application on NERSC's Cori system when using the Clang compiler, and a 15.7x speedup by switching max reductions to add reductions in the laplace mini-app when using the Cray-llvm compiler on Cori.

View PDFchevron_right
Chestnut: A GPU Programming Language for Non-Experts
Tia Newhall

2012

Graphics processing units (GPUs) are powerful devices capable of rapid parallel computation. GPU programming, however, can be quite difficult, limiting its use to experienced programmers and keeping it out of reach of a large number of potential users. We present Chestnut, a domainspecific GPU parallel programming language for parallel multidimensional grid applications. Chestnut is designed to greatly simplify the process of programming on the GPU, making GPU computing accessible to computational scientists who have little or no parallel programming experience, as well as a useful and powerful language for more experienced programmers. In addition, Chestnut has an optional GUI programming interface that makes GPU computing accessible to even novice programmers. Chestnut is intuitive and easy to use, while still powerful in the types of parallelism it can express. The language provides a single simple parallel construct that allows a Chestnut programmer to “think sequentially ” in e...

View PDFchevron_right
OpenMPC: Extended OpenMP Programming and Tuning for GPUs
Seyong Lee

2010

General-Purpose Graphics Processing Units (GPGPUs) are promising parallel platforms for high performance computing. The CUDA (Compute Unified Device Architecture) programming model provides improved programmability for general computing on GPGPUs. However, its unique execution model and memory model still pose significant challenges for developers of efficient GPGPU code. This paper proposes a new programming interface, called OpenMPC, which builds on OpenMP to provide an abstraction of the complex CUDA programming model and offers high-level controls of the involved parameters and optimizations. We have developed a fully automatic compilation and user-assisted tuning system supporting OpenMPC. In addition to a range of compiler transformations and optimizations, the system includes tuning capabilities for generating, pruning, and navigating the search space of compilation variants. Our results demonstrate that OpenMPC offers both programmability and tunability. Our system achieves 88% of the performance of the hand-coded CUDA programs.

View PDFchevron_right
Supporting multiple accelerators in high-level programming models
Chunhua Liao

Proceedings of the Sixth International Workshop on Programming Models and Applications for Multicores and Manycores - PMAM '15, 2015

Computational accelerators, such as manycore NVIDIA GPUs, Intel Xeon Phi and FPGAs, are becoming common in workstations, servers and supercomputers for scientific and engineering applications. Efficiently exploiting the massive parallelism these accelerators provide requires the designs and implementations of productive programming models. In this paper, we explore support of multiple accelerators in high-level programming models. We design novel language extensions to OpenMP to support offloading data and computation regions to multiple accelerators (devices). These extensions allow for distributing data and computation among a list of devices via easy-to-use interfaces, including specifying the distribution of multi-dimensional arrays and declaring shared data regions among accelerators. Computation distribution is realized by partitioning a loop iteration space among accelerators. We implement mechanisms to marshal/unmarshal and to move data of non-contiguous array subregions and shared regions between accelerators without involving CPUs. We design reduction techniques that work across multiple accelerators. Combined compiler and runtime support is designed to manage multiple GPUs using asynchronous operations and threading mechanisms. We implement our solutions for NVIDIA GPUs and demonstrate through example OpenMP codes the effectiveness of our solutions for performance improvement.

View PDFchevron_right
Productivity of GPUs under different programming paradigms
Maria Malik

Concurrency and Computation: Practice and Experience, 2012

Graphical processing units have been gaining rising attention because of their high performance processing capabilities for many scientific and engineering applications. However, programming such highly parallel devices requires adequate programming tools. Many such programming tools have emerged and hold the promise for high levels of performance. Some of such tools may require specialized parallel programming skills, while others attempt to target the domain scientist. The costs versus the benefits out of such tools are often unclear. In this work we examine the use of several of these programming tools such as Compute Unified Device Architecture, Open Compute Language, Portland Group Inc., and MATLAB in developing kernels from the (NAS) NASA Advanced Supercomputing parallel benchmarking suite. The resulting performance as well as the needed programmers' efforts were quantified and used to characterize the productivity of graphical processing units using these different programming paradigms.

View PDFchevron_right
A Programming Model for GPU-based Parallel Computing with Scalability and Abstraction
Balázs Domonkos

2009

In this paper, we present a multi-level programming model for recent GPU-based high performance computing systems. Involving cooperative stream threads and symmetric multiprocessing threads our model gives a computational framework that scales through multi-GPU environments to GPU-cluster systems. Instead of hiding the execution environment from the programmer using compiler extensions or metaprogramming techniques we aim a solution that both enables optimizations and provides abstract problem space mapping with code reusability and virtualization of hardware resources in order to decrease the programming effort. We evaluate an implementation of our model based on CUDA, OpenMP, and MPI2 technologies on a complex practical application scenario and discuss its performance scaling behavior.

View PDFchevron_right

References (40)

  1. Summary We can observe that based on the hardware performance AMD MI100 performs faster than NVIDIA V100 and slower than NVIDIA A100. The peak band- width percentage for the OpenMP programming model is 42.16%-94.68% for MI100 while it is at least 96% for the NVIDIA GPUs which demonstrates that AOMP needs further development. For Kokkos, the range is 74-99% for MI100, 72.87-99% for the NVIDIA GPUs, where the non-optimized version has lower References
  2. CSC LUMI supercomputer. https://www.lumi-supercomputer.eu/lumis-full- system-architecture-revealed/
  3. Frontier web page. https://www.olcf.ornl.gov/frontier/
  4. NVIDIA. CUDA. https://developer.nvidia.com/about-cuda
  5. Stone, J.E., Gohara, D., Shi, G.: OpenCL: a parallel programming standard for heterogeneous computing systems. In: Computing in Science & Engineering, vol. 12, no. 3, pp. 66-73, May-June 2010. https://doi.org/10.1109/MCSE.2010.69
  6. OpenMP Architecture Review Board. OpenMP Application Programming Inter- face, version 4.0. https://openmp.org/40pdf
  7. OpenACC Specification 3.0. https://www.openacc.org/sites/default/files/inline- images/Specification/OpenACC.3.0.pdf
  8. Davis, J.H., Daley, C., Pophale, S., Huber, T., Chandrasekaran, S., Wright, N.J.: Performance assessment of OpenMP compilers targeting NVIDIA V100 GPUs. In: Bhalachandra, S., Wienke, S., Chandrasekaran, S., Juckeland, G. (eds.) WACCPD 2020. LNCS, vol. 12655, pp. 25-44. Springer, Cham (2021). https://doi.org/10. 1007/978-3-030-74224-9 2
  9. Poenaru, A., Lin, W.-C., McIntosh-Smith, S.: A performance analysis of modern parallel programming models using a compute-bound application. In: 36th Inter- national Conference, ISC High Performance 2021, Frankfurt, Germany (2021)
  10. Khalilov, M., Timoveev, A.: Performance analysis of CUDA, OpenACC and OpenMP programming models on TESLA V100 GPU. In: Journal of Physics: Conference Series, vol. 1740 (2021)
  11. Deakin, T., McIntosh-Smith, S.: Evaluating the performance of HPC-style SYCL applications. In: Proceedings of the International Workshop on OpenCL (2020)
  12. Deakin, T., et al.: Performance portability across diverse computer architec- tures. In: 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pp. 1-13 (2019). https://doi.org/10.1109/ P3HPC49587.2019.00006
  13. AMD. ROCm Platform. https://github.com/RadeonOpenCompute/ROCm
  14. AMD. ROCm Documentation. https://rocmdocs.amd.com/en/latest/
  15. AMD. HIP. https://github.com/ROCm-Developer-Tools/HIP
  16. AMD. HIPify Tools. https://github.com/ROCm-Developer-Tools/HIPIFY
  17. AMD. HIP Porting Guide. https://github.com/RadeonOpenCompute/ROCm Documentation/blob/master/Programming Guides/HIP-porting-guide.rst
  18. CSC. Porting GPU Codes to HIP. https://github.com/csc-training/hip
  19. de Supinski, B.R., et al.: The ongoing evolution of OpenMP. In: Proceedings of the IEEE, vol. 106, no. 11, pp. 2004-2019, November 2018
  20. Khronos Group. SYCL 2020 Specification. https://www.khronos.org/registry/ SYCL/specs/sycl-2020/pdf/sycl-2020.pdf 20. Codeplay Software. ComputeCpp. https://www.codeplay.com/solutions/ ecosystem/
  21. Intel Corporation. SYCL* Compiler and Runtimes. https://github.com/intel/llvm
  22. Alpay, A., Heuveline, V.: SYCL beyond OpenCL: the architecture, current state and future direction of hipSYCL. In: Proceedings of the International Workshop on OpenCL (IWOCL 2020), Association for Computing Machinery, New York, Article vol. 8, no. 1 (2020). https://github.com/illuhad/hipSYCL
  23. triSYCL. https://github.com/trisycl/trisycl
  24. ORNL and Mentor Graphics. https://www.olcf.ornl.gov/2020/09/03/oak-ridge- leadership-computing-facility-fosters-gcc-compiler-development-with-mentor- contract/
  25. Denny, J.E., Lee, S. and Vetter, J.S.: Clacc: translating OpenACC to OpenMP in clang. In: 2018 IEEE/ACM 5th Workshop on the LLVM Compiler Infrastructure in HPC. LLVM-HPC), Dallas, TX, USA (2018)
  26. Clacc. https://github.com/llvm-doe-org/llvm-project/tree/clacc/main
  27. Zenker, E., et al.: Alpaka-an abstraction library for parallel kernel acceleration. In: 2016 IEEE International Parallel and Distributed Processing Symposium Work- shops (IPDPSW), pp. 631-640, May 2016
  28. Bussmann, M., et al.: Radiative signature of the relativistic kelvin-helmholtz insta- bility. In: SC 2013: Proceedings of the International Conference on High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-12 (2013)
  29. René, W., Sergei, B., Simeon, E., Jeffrey, K., Jan, S.: Cupla -C++ User interface for the Platform Independent Library alpaka. https://rodare.hzdr.de/record/1103
  30. Edwards, H.C., Trott, C.R., Sunderland, D.: Kokkos: enabling manycore perfor- mance portability through polymorphic memory access patterns. J Parall. Distrib. Comput. 74, 3202-3216 (2014). https://doi.org/10.1016/j.jpdc.2014.07.003
  31. AMD. hipfort. https://github.com/ROCmSoftwarePlatform/hipfort
  32. Deakin, T., Price, J., Martineau, M., McIntosh-Smith, S.: GPU-STREAM v2.0: benchmarking the achievable memory bandwidth of many-core processors across diverse parallel programming models. In: Paper presented at PˆMA Workshop at ISC High Performance, Frankfurt, Germany (2016). https://doi.org/10.1007/978- 3-319-46079-6 34
  33. Tom, D., Simon, M.-S.: BabelStream. https://github.com/UoB-HPC/ BabelStream
  34. CSC. Puhti Supercomputer. https://docs.csc.fi/computing/systems-puhti/
  35. CSC. Mahti Supercomputer. https://docs.csc.fi/computing/systems-mahti/
  36. CUPLA BabelStream Fork, v3.4-alpaka release https://github.com/jyoung3131/ BabelStream/releases/tag/v3.4-alpaka
  37. Konstantinidis, E., Cotronis, Y.: A quantitative roofline model for GPU kernel performance estimation using micro-benchmarks and hardware metric profiling. J. Parall. Distrib. Comput. 107, 37-56 (2017)
  38. Mixbench. https://github.com/ekondis/mixbench
  39. Reproduce the results of the paper Evaluating GPU Programming Models for the LUMI Supercomputer. https://zenodo.org/record/6307447
  40. Elbencho. https://github.com/breuner/elbencho

Related papers

GPU programming in a high level language
Vijay Saraswat

Proceedings of the 2011 ACM SIGPLAN X10 Workshop, 2011

GPU architectures have emerged as a viable way of considerably improving performance for appropriate applications. Program fragments (kernels) appropriate for GPU execution can be implemented in CUDA or OpenCL and glued into an application via an API. While there is plenty of evidence of performance improvements using this approach, there are many issues with productivity. Programmers must understand an additional programming model and API to program the accelerator; concurrency and synchronization in this programming model is typically expressed differently from the programming model for the host. On top of this, the languages used to write kernels are very low level and thus prone to the kinds of errors that one does not encounter in higher level languages. Programmers must explicitly deal with moving data back-and-forth between the host and the accelerator. These problems are compounded when the user code must be run across a cluster of accelerated nodes. Now the host programming model must further be extended with constructs to deal with scale-out and remote accelerators. We believe there is a critical need for a single source programming model that can be used to write clean, efficient code for heterogeneous, multi-core and scale-out architectures. The APGAS programming model has been developed for such architectures over the past six years. APGAS is based on four fundamental (and architecture-independent) notions: locality, asynchrony, conditional atomicity and order. X10 is an instantiation of the APGAS programming model on top of a base sequential language with Java-style productivity. Earlier work has shown that X10 can be used to write clean and efficient code for homogeneous multi-cores, SMPs, Cell-accelerated nodes, and clusters of such nodes. In this paper we show how X10 programmers can write code that can be compiled and run on GPUs. GPU programming idioms such as threads, blocks, barriers, constant memory, local registers, shared memory variables, etc. can be directly expressed in X10, and do not require new language extensions. We present the design of an extension of the X10-to-C++ compiler which recognizes such idioms and produces CUDA kernel code. We show several benchmarks written in this style. The performance of these kernels is within 80% of handwritten CUDA kernels. [Copyright notice will appear here once 'preprint' option is removed.] We believe these results establish X10 as a single-source programming language in which clean, efficient programs can be written for GPU-accelerated clusters.

View PDFchevron_right
An extension of the StarSS programming model for platforms with multiple GPUs
R. Mayo, Eduard Ayguade

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2009

While general-purpose homogeneous multi-core architectures are becoming ubiquitous, there are clear indications that, for a number of important applications, a better performance/power ratio can be attained using specialized hardware accelerators. These accelerators require specific SDK or programming languages which are not always easy to program. Thus, the impact of the new programming paradigms on the programmer's productivity will determine their success in the highperformance computing arena. In this paper we present GPU Superscalar (GPUSs), an extension of the Star Superscalar programming model that targets the parallelization of applications on platforms consisting of a general-purpose processor connected with multiple graphics processors. GPUSs deals with architecture heterogeneity and separate memory address spaces, while preserving simplicity and portability. Preliminary experimental results for a well-known operation in numerical linear algebra illustrate the correct adaptation of the runtime to a multi-GPU system, attaining notable performance results.

View PDFchevron_right
Early evaluation of directive-based GPU programming models for productive exascale computing
Seyong Lee

2012 International Conference for High Performance Computing, Networking, Storage and Analysis, 2012

Graphics Processing Unit (GPU)-based parallel computer architectures have shown increased popularity as a building block for high performance computing, and possibly for future Exascale computing. However, their programming complexity remains as a major hurdle for their widespread adoption. To provide better abstractions for programming GPU architectures, researchers and vendors have proposed several directive-based GPU programming models. These directive-based models provide different levels of abstraction, and required different levels of programming effort to port and optimize applications. Understanding these differences among these new models provides valuable insights on their applicability and performance potential. In this paper, we evaluate existing directive-based models by porting thirteen application kernels from various scientific domains to use CUDA GPUs, which, in turn, allows us to identify important issues in the functionality, scalability, tunability, and debuggability of the existing models. Our evaluation shows that directive-based models can achieve reasonable performance, compared to hand-written GPU codes.

View PDFchevron_right
Directive-Based Compilers for GPUs
Ruben Gran Tejero

General Purpose Graphics Computing Units can be effectively used for enhancing the performance of many contemporary scientific applications. However, programming GPUs using machine-specific notations like CUDA or OpenCL can be complex and time consuming. In addition, the resulting programs are typically fine-tuned for a particular target device. A promising alternative is to program in a conventional and machine-independent notation extended with directives and use compilers to generate GPU code automatically. These compilers enable portability and increase programmer productivity and, if effective, would not impose much penalty on performance. This paper evaluates two such compilers, PGI and Cray. We first identify a collection of standard transformations that these compilers can apply. Then, we propose a sequence of manual transformations that programmers can apply to enable the generation of efficient GPU kernels. Lastly, using the Rodinia Benchmark suite, we compare the performance of the code generated by the PGI and Cray compilers with that of code written in CUDA. Our evaluation shows that the code produced by the PGI and Cray compilers can perform well. For 6 of the 15 benchmarks that we evaluated, the compiler generated code achieved over 85% of the performance of a hand-tuned CUDA version.

View PDFchevron_right
ÆminiumGPU: An Intelligent Framework for GPU Programming
Bruno Cabral

Lecture Notes in Computer Science, 2013

As a consequence of the immense computational power available in GPUs, the usage of these platforms for running data-intensive general purpose programs has been increasing. Since memory and processor architectures of CPUs and GPUs are substantially different, programs designed for each platform are also very different and often resort to a very distinct set of algorithms and data structures. Selecting between the CPU or GPU for a given program is not easy as there are variations in the hardware of the GPU, in the amount of data, and in several other performance factors. AEminiumGPU is a new data-parallel framework for developing and running parallel programs on CPUs and GPUs. AEminiumGPU programs are written in a Java using Map-Reduce primitives and are compiled into hybrid executables which can run in either platforms. Thus, the decision of which platform is going to be used for executing a program is delayed until run-time and automatically performed by the system using Machine-Learning techniques. Our tests show that AEminiumGPU is able to achieve speedups up to 65x and that the average accuracy of the platform selection algorithm, in choosing the best platform for executing a program, is above 92%.

View PDFchevron_right

Related topics

  • Computer Science
  • Parallel Computing
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved