A Unified Compiler Framework for Work and Data Placement

2012

We propose a new theoretical model for parallelism. The model is explictly based on data and work distributions, a feature missing from other theoretical models. The major theoretic result is that data movement can then be derived by formal reasoning. While the model has an immediate interpretation in distributed memory parallelism, we show that it can also accomodate shared memory and hybrid architectures such as clusters with accelerators. The model gives rise in a natural way to objects appearing in widely different parallel programming systems such as the PETSc library or the Quark task scheduler. Thus we argue that the model offers the prospect of a high productivity programming system that can be compiled down to proven high-performance environments.

ACM Queue, 2014

In 2002 DARPA (Defense Advanced Research Projects Agency) launched a major initiative in HPCS (high-productivity computing systems). The program was motivated by the belief that the utilization of the coming generation of parallel machines was gated by the difficulty of writing, debugging, tuning, and maintaining software at peta scale. As part of this initiative, DARPA encouraged work on new programming languages, runtimes, and tools. It believed that by making the expression of parallel constructs easier, matching the runtime models to the heterogeneous processor architectures under development, and providing powerful integrated development tools, it might improve programmer productivity. This is a reasonable conjecture, but we sought to go beyond conjecture to actual measurements of productivity gains. While there is no established method for measuring programmer productivity, it is clear that a productivity metric must take the form of a ratio: programming results achieved over the cost of attaining them. In this case, results are defined as successfully creating a set of parallel programs that ran correctly on two workstation cores. This is a long way from peta scale, but since new parallel software often starts out this way (and is then scaled and tuned on ever-larger numbers of processors), we viewed it as a reasonable approximation. Moreover, results found with two cores should be of interest to those coding nearly any parallel application, no matter how small. Cost was simpler to handle once results was defined, since it could reasonably be approximated by the time it took to create this set of parallel programs. The purpose of this study was to measure programmer productivity, thus defined, over several years starting in 2002, the beginning of the HPCS initiative. The comparison was primarily focused on two approaches to parallel programming: the SPMD (single program multiple data) model as exemplified by C/MPI (message-passing interface), and the APGAS (asynchronous partitioned global address space) model supported by new languages such as X10 (http://x10-lang.org), although differences in environment and tooling were also studied. Note that the comparison was not between C/MPI as it has come to be and X10 as it is now. Rather, it was a historical contrast of the way things were in 2002 with the way things are now. Indeed, C++ with its exceptions and MPI-2 with its onesided communication protocol likely enhance programmer productivity and are worthy of study in their own right. Given our objective, we sought to replicate as closely as possible the programming environment found in 2002 for users of C/MPI. This included the gdb debugger, along with a typical set of command-line tools. For X10, we used Eclipse (http://www.eclipse.org) with the X10 plug-in as it was found in 2010, the date of this study. X10 was developed as part of the HPCS initiative. It combines

1994

The principal goal of this chapter is to introduce the common issues that a programmer faces when implementing a parallel application. The treatment assumes that the reader is familiar with programming a uniprocessor using a conventional language, such as Fortran. The principal challenge of parallel programming is to decompose the program into subcomponents that can be run in parallel. However, to understand some of the low-level issues of decomposition, the programmer must have a simple view of parallel machine architecture. Thus we begin our treatment with a discussion of this topic. This discussion, found in Section 2.2, focuses on two main parallel machine organizations — shared memory and distributed memory — that characterize most current machines. The section also treats clusters, which are hybrids of the two main memory designs. The standard parallel architectures support a variety of decomposition strategies, such as decomposition by task (task parallelism) and decompositio...

Computer, 2000

Lack of appropriate abstractions makes programming for parallel architectures more difficult than writing sequential software. Mentat addresses this problem by extending C++ to include parallelism encapsulation. May 1993 WO problems plague programming for parallel multiple-instruction, multiple-data (MIMD) architectures. First, writing parallel programs by hand is very difficult. The programmer must manage communication, synchronization, and scheduling of tens to thousands of independent processes. Correctly managing the environment requires considerable time and energy and often overwhelms the programmer. Second, code implemented on a particular architecture is seldom usable on other MIMD architectures, since the tools, techniques, and library facilities to parallelize the application are platform specific. Thus, porting the application to a new architecture requires considerable effort. Given the plethora of new architectures and the rapid obsolescence of existing architectures, this represents a continuing time investment. Mentat, an object-oriented parallel processing system that I developed with my students at the University of Virginia, directly addresses the difficulty of programming MIMD architectures and the portability of applications. Its three primary design objectives are to provide (1) easy-to-use parallelism, (2) high performance via parallel execution, and (3) application portability across a wide range of platforms. The underlying premise is that writing programs for parallel machines does not have to be difficult. Instead, it's the lack of appropriate abstractions that has made parallel architectures difficult to program and kept them inaccessible to mainstream production-system programmers. The Mentat philosophy on parallel computing is guided by two observations. The first is that the programmer understands the application domain and can make better data and computation partitioning decisions than the compiler. The truth of this is evidenced by the fact that most successful production parallel applications have been hand-coded using low-level primitives. In these applications the programmer has decomposed and distributed both the data and the computation. The second observation is that management of tens to thousands of asynchronous tasks, where timing-dependent errors are easy to make, is beyond the capacity of most programmers. Compilers, on the other hand, are very good at ensuring that events happen in the right order and can more readily and correctly manage communication and synchronization, particularly in highly asynchronous, non-SPMD (single program, multiple data) environments.

The process of writing parallel programs is not as simple or common as traditional sequential programming. A programmer must have an in-depth knowledge of the program at hand, as well as the resources available when the program is executed. By designing a compiler that can automatically parallelise sequentially written programs, is obviously of great benefit. The current efforts concentrate mainly on fine grained parallelism. With the move towards using clusters of workstations as platforms for parallel processing, coarse grained parallelism is becoming a very important research issue. This paper reports on the development of a simple parallelising compiler exploiting course grained parallelism in sequential programs. In particular, the use of three structured algorithms used to divide the parallelisation process into manageable phases is presented.

Parallel Computing, 2013

The widespread use of multicore processors is not a consequence of significant advances in parallel programming.

Journal of Parallel and Distributed Computing, 1998

2004

We describe a set of language extensions to Java to support parallel programming with distribution annotations. The system provides an integrated system of placement annotations on both code and data. This allows the programmer to freely mix data-parallel programming similar to HPF, and taskparallel programming similar to OpenMP.

Parallel Processing Letters, 2000

Traditionally, the compilation of data-parallel languages is targeted to low-level runtime environments: abstract processors are mapped onto static system processes, which directly address the low-level IPC library. Alternatively, we propose to map each HPF abstract processor onto a "lightweight process" (thread) which can be freely migrated between nodes together with the data it manages, under the supervision of some external scheduler. We discuss the pros and cons of such an approach and the facilities which must be provided by the multithreaded runtime. We describe a prototype HPF compiler built along these lines, based on the Adaptor HPF compiler and the PM2 multithreaded runtime environment.