Challenges in Fluid Flow Simulations Using Exascale Computing

Computing in Science & Engineering, 2012

Unsteady computational fluid dynamics simulations of turbulence are performed using up to 64 graphics processors. The results from two GPU clusters and a CPU cluster are compared. A second-order staggered-mesh spatial discretization is coupled with a low storage three-step Runge-Kutta time advancement and pressure projection at each substep. The pressure Poisson equation dominates the solution time and is solved with the preconditioned Conjugate Gradient method. The CFD algorithm is optimized to use the fast shared-memory on the GPUs and to use communication/computation overlapping. Detailed timings reveal that the internal calculations now occur so efficiently that the operations related to communication are the scaling bottleneck at all but the very largest problem sizes that can fit on the hardware. 2. Implementation 2.1 Hardware We will primarily discuss results computed on the Lincoln supercomputer housed at NCSA. This machine has 96 Tesla S1070s (384 GPUs total). Each GPU has 4GB of memory and a theoretical bandwidth of 102 GBs to that memory. Each GPU has a 4x PCI-e Gen2 connection (2 GB/s) to its CPU host. In addition, we will also perform tests on Orion, which is an in-house GPU machine containing 4 NVIDIA 295 cards, 8 GPUs. On Orion, each GPU has 0.9 GB of memory and a theoretical bandwidth of 112 GBs. The connection between the GPUs on Orion uses a 8x PCI-e Gen2 connection (4 GB/s) and for simplicity communication still uses the MPI protocol. Also we replaced the first and second GPUs with GTX 480 and Tesla C2070 cards in order to run some cases with these new GPUs. These GPU results will be compared to the CPU cores on Lincoln (quad core Intel 64 Harpertown). The low-level CFD algorithm structure is dictated by two key features of the GPU hardware. First, the GPUs read/write memory is one order of magnitude faster when the memory is read linearly. Random reads/writes are comparatively slow on a GPU. In addition, each multi-processor on the GPU has some very fast on-chip memory (shared memory) which serves essentially as an addressable programsupervised cache. CFD, like all most three-dimensional PDE solution applications, requires considerable random memory accesses (even when using structured meshes). Roughly 90% of these slow random memory accesses can be eliminated by: (1) linearly reading large chunks of data into the shared-memory space, which is fast for all accesses, (2) operating on the data in the shared-memory, and then (3) writing the processed data back to the main GPU memory (global memory) linearly. This optimization is the key to the 45x speedup of the GPU over a CPU.

2005

In parallel computing, scalability is an important issue and getting linear speed-ups is difficult for most codes. Superlinear speed-up has been achieved on an eight-processor Flosolver system for a multigrid Navier-Stokes code. The physical problem solved, the parallelization method, the speed-ups obtained and possible explanations for this result are discussed here. PARALLEL computers are used nowadays for all non-trivial Computational Fluid Dynamics (CFD) calculations. The need for large computations makes parallel computers necessary. Also, since a majority of CFD codes use finite difference or finite volume methods, the domain decomposition technique can be used for parallelization, which results in good parallel efficiencies. Other codes that use spectral techniques where global communication is required are known to have moderate to poor scalability. The multigrid technique which is used to accelerate the convergence of Poisson solvers is generally considered to hamper parallel efficiency due to its global dependencies 1,2. The most commonly used metric to measure scalability is the speed-up, which is defined as the ratio of the time taken by the sequential code to that of the parallel code. Speed-up is a function of the number of processors used. Parallel efficiency is defined as speed-up divided by the number of processors and in general, falls off as the number of processors increases. At the Flosolver laboratory, we have achieved a superlinear speed-up of 11 on an eight-processor system using the FloSwitch 3 for communication on a multigrid laminar Navier-Stokes code written at the Jawaharlal Nehru Centre for Advanced Scientific Research, Bangalore. Recently, Dresden 4 reported a speed-up of nearly 140 with a 120-processor system on a finite element Navier-Stokes code. Here we document briefly the Flosolver experience.

2004

Advances in computational science are closely tied to developments in highperformance computing. We consider the case of shelf sea modelling where models have been growing in complexity and where model domains have been growing and grid resolutions shrinking in pace with the increasing storage capacity and computing power of high-end systems. Terascale systems are now readily available with performance levels measurable in TeraFlop/s and memories counted in TeraBytes. The scientific case is now being made for regional models at 1km resolution, allowing the accurate representation of eddies, fronts and other regions containing steep gradients. The hydrodynamic model is increasingly being coupled with other models in multidisciplinary studies e.g. ecosystem modelling and wave modelling. We show that the performance attainable from the POLCOMS hydrodynamic code is measurable at about 0.5 TeraFlop/s on an IBM p690 cluster with 1024 processors. The scalability on this system and others is excellent up to 1000 processors. We describe a wide range of optimisations which have together enabled this code to reach these performance levels.

Computers & Fluids, 1990

Anais Estendidos do IX Simpósio Brasileiro de Arquitetura de Computadores e Processamento de Alto Desempenho (SBAC-PAD Estendido 1997)

Using the same numerical and parallelization strategy, high performance pseudo-spectral methods for the simulation of turbulent flows, two-dimensional two-phase flows, and flows in porous media were developed. The codes are designed to perform high resolution, space and time accurate simulations of complex flows on various computer architectures, including distributed memory systems. The codes give good speedups on parallel machines, and good performance on vector machines.

Proceedings of the 8th Annual ACM India Conference, 2015

In this paper we present a comparison study of two widely used parallelizing frameworks on the CPU, namely, OpenMP and Intel Threading Building Blocks (TBB). The particular problem domain we apply to is a grid-based fluid simulation solver. The standard Eulerian grid-based fluid solver discretizes the Navier-Stokes equation on a staggered but regular grid and computes the fluid parameters like velocity and pressure in each grid cell. We use OpenMP and TBB to parallelize this computation, and study the behaviour of our implementation on each framework, while working with different number of threads and CPU cores. We provide arguments in support of implementing a mixed solution strategy using both the parallelizing frameworks together, thus improving performance over when either is used in isolation.

2012

Computational fluid dynamics is an increasingly important application domain for computational scientists. In this paper, we propose and analyze optimizations necessary to run CFD simulations consisting of multibillion-cell mesh models on large processor systems. Our investigation leverages the general industrial Navier-Stokes CFD application, Code_Saturne, developed by Electricité de France for incompressible and nearly compressible flows. In this paper, we outline the main bottlenecks and challenges for massively parallel systems and emerging processor features such as many-core, transactional memory, and thread level speculation. We also present an approach based on an octree search algorithm to facilitate the joining of mesh parts and to build complex larger unstructured meshes of several billion grid cells. We describe two parallel strategies of an algebraic multigrid solver and we detail how to introduce new levels of parallelism based on compiler directives with OpenMP, transactional memory and thread level speculation, for finite volume cell-centered formulation and face-based loops. A renumbering scheme for mesh faces is proposed to enhance thread-level parallelism. and implementations capable of simulating with multibillions of cells or particles are beginning to emerge within the research community. Nevertheless, one of the bigger challenges is to reach this capability with general CFD Navier-Stokes industrial software.

2015

Computational Fluid Dynamics (CFD) simulations are aimed to reconstruct the reality of fluid motion and behaviour as accurately as possible, to better understand the natural phenomena under specified conditions. Ideally, computational models would need to cover different scales and geometric configurations, and the classic CFD solvers most often require long computational times to satisfy the convergence criteria. With the advent of heterogeneous compute platforms (including Graphics Processing Units GPUs), CFD algorithms can now be implemented to give results in near real-time. The current paper briefly reviews and demonstrates in a general way, two methods able to harness the power of GPUs, to speed up numerical simulations of fluid flows for industrial applications. These include the Highly Simplified Marker and Cell Method (HSMAC), and Lattice-Boltzmann Method (LBM) implemented on GPUs using OpenCL. This paper describes general capabilities for compute and graphics, and method p...

Concurrency and Computation: Practice and Experience, 2007

Direct numerical simulation (DNS) of turbulent flows is widely recognized to demand fine spatial meshes, small timesteps, and very long run-times to properly resolve the flow field. To overcome these limitations, most DNS is performed on supercomputing machines. With the rapid development of terascale (and, eventually, petascale) computing on thousands of processors, it has become imperative to consider the development of DNS algorithms and parallelization methods that are capable of fully exploiting these massively parallel machines. A highly parallelizable algorithm for the simulation of turbulent channel flow that allows for efficient scaling on several thousand processors is presented. A model that accurately predicts the performance of the algorithm is developed and compared with experimental data. The results demonstrate that the proposed numerical algorithm is capable of scaling well on petascale computing machines and thus will allow for the development and analysis of high Reynolds number channel flows.

Computers & Fluids, 2011

This paper investigates the use of TELEMAC (a Finite Element-based hydrodynamics suite) on massively parallel computer architectures. The performance of TELEMAC is illustrated using two separate test cases. The first considers the use of TELEMAC-2D for simulating tidal currents in the vicinity of a renewable energy marine turbine farm, in order to provide reliable estimates of the expected energy yield. The second demonstrates the use of TELEMAC-3D for assessing the effects of fresh water discharges on the salinity distribution in a coastal lagoon. The simulations have been performed with meshes ranging from 2 to 12 million elements, and good scaling performance is achieved on a variety of different computer architectures.