Data Parallelism

This paper describes and evaluates three architectural methods for accomplishing data parallel computation in a programmable embedded system. Comparisons are made between the well-studied Very Long Instruction Word (VLIW) and Single Instruction Multiple Packed Data (SIM pD) paradigms; the less-common Single Instruction Multiple Disjoint Data (SIM d D) architecture is described and evaluated. A taxonomy is defined for data-level parallel architectures, and patterns of data access for parallel computation are studied, with measurements presented for over 40 essential telecommunication and media kernels. While some algorithms exhibit data-level parallelism suited to packed vector computation, it is shown that other kernels are most efficiently scheduled with more flexible vector models. This motivates exploration of non-traditional processor architectures for the embedded domain.

This paper describes and evaluates three architectural methods for accomplishing data parallel computation in a programmable embedded system. Comparisons are made between the well-studied Very Long Instruction Word (VLIW) and Single Instruction Multiple Packed Data (SIM pD) paradigms; the less-common Single Instruction Multiple Disjoint Data (SIM d D) architecture is described and evaluated. A taxonomy is defined for data-level parallel architectures, and patterns of data access for parallel computation are studied, with measurements presented for over 40 essential telecommunication and media kernels. While some algorithms exhibit data-level parallelism suited to packed vector computation, it is shown that other kernels are most efficiently scheduled with more flexible vector models. This motivates exploration of non-traditional processor architectures for the embedded domain.

One well-liked method for condensing massive language models (LLMs) into smaller, faster, more effective versions without sacrificing performance is knowledge distillation (KD). However, it is no longer feasible to run distillation on a single device as LLMs grow to hundreds of billions of parameters; it is simply too computationally demanding. In this paper, we investigate how to leverage multi-GPU configurations to make KD scale. To overcome communication bottlenecks and accelerate training, our method combines adaptive gradient compression with tensor, pipeline, and data parallelism. Tested on transformer-based LLMs, our framework maintains strong accuracy for both understanding and generation tasks, reduces communication overhead by 27%, and provides up to 3.4× faster training than single-GPU baselines.

Task-based libraries such as Intel's Threading Building Blocks (TBB) provide higher levels of abstraction than threads for parallel programming. Work remains, however, to determine how straightforward it is to use these libraries to express various patterns of parallelism. This case study focuses on a particular pattern: pipeline parallelism. We attempted to transform three representative applicationscontent-based image retrieval, compression and video encoding -to pipelines using TBB. We successfully converted two of the three applications. In the successful cases we discuss our transformation process and contrast the expressivity and performance of our implementations to existing Pthreads versions; in the unsuccessful case, we detail what the challenges were and propose potential solutions.

This paper describes a numerical method for the parallel solution of the differential measure inclusion problem posed by mechanical multibody systems containing bilateral and unilateral frictional constraints. The method proposed has been implemented as a set of parallel algorithms leveraging NVIDIA’s Compute Unified Device Architecture (CUDA) library support for multi-core stream computing. This allows the proposed solution to run on a wide variety of GeForce and TESLA NVIDIA graphics cards for high performance computing. Although the methodology relies on the solution of cone complementarity problems known to be fine-grained in terms of data dependency, a suitable approach has been developed to exploit parallelism with low overhead in terms of memory access and thread synchronization. Additionally, a parallel collision detection algorithm has been incorporated to further exploit available parallelism. Initial numerical tests described in this paper demonstrate a speedup of one ord...

With current systems, some important complex queries may take days to complete because of: (1) the volume of data to be processed, (2) limited aggregate resources. Introducing parallelism addresses the first problem. Cheaper, but powerful computing resources solve the second problem. According to a survey by Brodie.' only 10% of computerized data is in data bases. This is an argument for both more variety and volume of data to be moved into data base systems. We conjecture that the primary reasons for this low percentage are that data base management systems (DBMSs) still need to provide far greater functionality and improved performance compared to a combination of application programs and file systems. This paper addresses the issues and solutions relating to intraquery parallelism in a relational DBMS supporting SQL. Instead of focussing only on a few algorithms for a subset of the problems, we provide a broad framework for the study of the numerous issues that need to be addressed in supporting parallelism efficiently and flexibly. We also discuss the impact that parallelization of complex queries has on short transactions which have stringent response time constraints. The pros and cons of the shared nothing. shared disks and shared everything architectures for parallelism are enumerated. The impact of parallelism on a number of components of an industrialstrength DBMS are pointed out. The different stages of query processing during which parallelism may be gainfully employed are identified. The interactions between parallelism and the traditional sys-tems' pipelining technique are analyzed. Finally, the performance implications of parallelizing a specific co,nplex query are studied. This gives us a range cf sample points for different parameters of a parallel system architecture, namely, I/O and communication bandwidth as a function of aggregate MIPS.

Emerging computing architectures such as near-memory computing (NMC) promise improved performance for applications by reducing the data movement between CPU and memory. However, detecting such applications is not a trivial task. In this ongoing work, we extend the state-of-the-art platform-independent software analysis tool with NMC related metrics such as memory entropy, spatial locality, data-level, and basic-block-level parallelism. These metrics help to identify the applications more suitable for NMC architectures. • Software and its engineering → Dynamic analysis.

Automatic optimization of application-specific instruction-set processor (ASIP) architectures mostly focuses on the internal memory hierarchy design, or the extension of reduced instruction-set architectures with complex custom operations. This paper focuses on very long instruction word (VLIW) architectures and, more specifically, on automating the selection of an application specific VLIW issue-width. The issuewidth selection strongly influences all the important processor properties (e.g. processing speed, silicon area, and power consumption). Therefore, an accurate and efficient issue-width estimation and optimization are some of the most important aspects of VLIW ASIP design. In this paper, we first compare different methods for the estimation of required the issue-width, and subsequently introduce a new force-based parallelism estimation method which is capable of estimating the required issue-width with only 3% error on average. Furthermore, we present and compare two techniques for estimating the required issue-width of software pipelined loop kernels and show that a simple utilization-based measure provides an error margin of less than 1% on average.

SIMD (single instruction multiple data)-type processors have been found very efficient in image processing applications, because their repetitive structure is able to exploit the huge amount of data-level parallelism in pixel-type operations, operating at a relatively low energy consumption rate. However, current SIMD architectures lack support for dynamic communication between processing elements, which is needed to efficiently map a set of non-linear algorithms. An architecture for dynamic communication support has been proposed, but this architecture needs large amounts of buffering to function properly. In this paper, three architectures supporting dynamic communication without the need of large amounts of buffering are presented, requiring 98% less buffer space. Cycle-true communication architecture simulators have been developed to accurately predict the performance of the different architectures. Simulations with several test algorithms have shown a performance improvement of up to 5x compared to a locally connected SIMDprocessor. Also, detailed area models have been developed, estimating the three proposed architectures to have an area overhead of 30-70% compared to a locally connected SIMD architecture (like the IMAP). When memory is taken into account as well, the overhead is estimated to be 13-28%.

Despite the various research initiatives and proposed programming models, efficient solutions for parallel programming in HPC clusters still rely on a complex combination of different programming models (e.g., OpenMP and MPI), languages (e.g., C++ and CUDA), and specialized runtimes (e.g., Charm++ and Legion). On the other hand, task parallelism has shown to be an efficient and seamless programming model for clusters. This paper introduces OpenMP Cluster (OMPC), a task-parallel model that extends OpenMP for cluster programming. OMPC leverages OpenMP's offloading standard to distribute annotated regions of code across the nodes of a distributed system. To achieve that it hides MPI-based data distribution and load-balancing mechanisms behind OpenMP task dependencies. Given its compliance with OpenMP, OMPC allows applications to use the same programming model to exploit intra-and internode parallelism, thus simplifying the development process and maintenance. We evaluated OMPC using Task Bench, a synthetic benchmark focused on task parallelism, comparing its performance against other distributed runtimes. Experimental results show that OMPC can deliver up to 1.53x and 2.43x better performance than Charm++ on CCR and scalability experiments, respectively. Experiments also show that OMPC performance weakly scales for both Task Bench and a real-world seismic imaging application.

Codes written in a naive way seldom effectively exploit the computing resources, while writing optimized codes is usually a complex task that requires certain levels of expertise. This problem is further increased in the presence of heterogeneous devices, which present more tunable parameters than regular CPUs and high sensitivity to the optimization decisions taken. Furthermore, portability is an added concern given the wide variety of accelerators available. This paper tackles this problem adding an automatic optimizer to a library that already provides an easy and portable way to program heterogeneous devices, the Heterogeneous Programming Library (HPL). Our optimizer takes as input a simple version of a code and then tunes it for the device where it is going to be executed by performing the most usual set of optimizations applicable in heterogeneous devices. These optimizations are parametrized using a set of optimization parameters that need to be tuned for the device. The HPL library has also been equipped with an autotuner that can be used to this purpose. The effectiveness of the autotuner and the optimizer has been tested on several codes and devices. The results show that the combination of the autotuner and the optimizer make the tested codes 16 times faster on average than the original codes written by the programmer.

Heterogeneous devices require much more work from programmers than traditional CPUs, particularly when there are several of them, as each one has its own memory space. Multidevice applications require to distribute kernel executions and, even worse, arrays portions that must be kept coherent among the different device memories and the host memory. In addition, when devices with different characteristics participate in a computation, optimally distributing the work among them is not trivial. In this paper we extend an existing framework for the programming of accelerators called Heterogeneous Programming Library (HPL) with three kinds of improvements that facilitate these tasks. The first two ones are the ability to define subarrays and subkernels, which distribute kernels on different devices. The last one is a convenient extension of the subkernel mechanism to distribute computations among heterogeneous devices seeking the best work balance among them. This last contribution includes two analytical models that have proved to automatically provide very good work distributions. Our experiments also show the large programmability advantages of our approach and the negligible overhead incurred.

Multicore machines are becoming common. There are many languages, language extensions and libraries devoted to improve the programmability and performance of these machines. In this paper we compare two libraries, that face the problem of programming multicores from two different perspectives, task parallelism and data parallelism. The Intel Threading Building Blocks (TBB) library separates logical task patterns, which are easy to understand, from physical threads, and delegates the scheduling of the tasks to the system. On the other hand, Hierarchically Tiled Arrays (HTAs) are data structures that facilitate locality and parallelism of array intensive computations with a block-recursive nature following a data-parallel paradigm. Our comparison considers both ease of programming and the performance obtained using both approaches. In our experience, HTA programs tend to be smaller or as long as TBB programs, while performance of both approaches is very similar.

Machine learning can provide deep insights into data, allowing machines to make high-quality predictions and having been widely used in real-world applications, such as text mining, visual classification, and recommender systems. However, most sophisticated machine learning approaches suffer from huge time costs when operating on large-scale data. This issue calls for the need of Largescale Machine Learning (LML), which aims to learn patterns from big data with comparable performance efficiently. In this paper, we offer a systematic survey on existing LML methods to provide a blueprint for the future developments of this area. We first divide these LML methods according to the ways of improving the scalability: 1) model simplification on computational complexities, 2) optimization approximation on computational efficiency, and 3) computation parallelism on computational capabilities. Then we categorize the methods in each perspective according to their targeted scenarios and introduce representative methods in line with intrinsic strategies. Lastly, we analyze their limitations and discuss potential directions as well as open issues that are promising to address in the future.

Parallel Machine (APM) model separates the definitions of parallel operations from the application algorithm, which defines the sequence of parallel operations to be executed. An APM contains a set of parallel operation definitions, which specify how the computation is organized into independent sites of computation and what data exchanges are required. This paper adds explicit cost models as the third component of an APM system. The costs of parallel operations can be obtained either by analyzing a parallel operation definition, or by measuring performance on a real machine. Costs with monotonicity constraints allow the cost of an algorithm to be transformed automatically as the algorithm itself is transformed.

SystemML aims at declarative, large-scale machine learning (ML) on top of MapReduce, where high-level ML scripts with R-like syntax are compiled to programs of MR jobs. The declarative specification of ML algorithms enables---in contrast to existing large-scale machine learning libraries---automatic optimization. SystemML's primary focus is on data parallelism but many ML algorithms inherently exhibit opportunities for task parallelism as well. A major challenge is how to efficiently combine both types of parallelism for arbitrary ML scripts and workloads. In this paper, we present a systematic approach for combining task and data parallelism for large-scale machine learning on top of MapReduce. We employ a generic Parallel FOR construct (ParFOR) as known from high performance computing (HPC). Our core contributions are (1) complementary parallelization strategies for exploiting multi-core and cluster parallelism, as well as (2) a novel cost-based optimization framework for auto...

With the increasing demand for deep learning in the last few years, CNNs have been widely used in many applications and have gained interest in classification, regression, and image recognition tasks. The training of these deep neural networks is compute-intensive and takes days or even weeks to train the model from scratch. The compute-intensive nature of these deep neural networks sometimes limits the practical implementation of CNNs in real-time applications. Therefore, the computational speedup in these networks is of utmost importance, which generates interest in CNN training acceleration. Much research is going on to meet the computational requirement and make it feasible for real-time applications. Because of its simplicity, data parallelism is used primarily, but it performs badly sometimes. In most cases, researchers prefer model parallelism to data parallelism, but it is not always the best choice. Therefore, in this study, we implement a hybrid of both data and model para...

In this paper, we investigate how to exploit task-parallelism during the execution of the Cholesky factorization on clusters of multicore processors with the SMPSs programming model. Our analysis reveals that the major difficulties in adapting the code for this operation in ScaLAPACK to SMPSs lie in algorithmic restrictions and the semantics of the SMPSs programming model, but also that they both can be overcome with a limited programming effort. The experimental results report considerable gains in performance and scalability of the routine parallelized with SMPSs when compared with conventional approaches to execute the original ScaLAPACK implementation in parallel as well as two recent message-passing routines for this operation. In summary, our study opens the door to the possibility of reusing messagepassing legacy codes/libraries for linear algebra, by introducing up-to-date techniques like dynamic out-of-order scheduling that significantly upgrade their performance, while avoiding a costly rewrite/reimplementation.

Background: Sequence similarity searching is an important and challenging task in molecular biology and next-generation sequencing should further strengthen the need for faster algorithms to process such vast amounts of data. At the same time, the internal architecture of current microprocessors is tending towards more parallelism, leading to the use of chips with two, four and more cores integrated on the same die. The main purpose of this work was to design an effective algorithm to fit with the parallel capabilities of modern microprocessors. A parallel algorithm for comparing large genomic banks and targeting middle-range computers has been developed and implemented in PLAST software. The algorithm exploits two key parallel features of existing and future microprocessors: the SIMD programming model (SSE instruction set) and the multithreading concept (multicore). Compared to multithreaded BLAST software, tests performed on an 8-processor server have shown speedup ranging from 3 to 6 with a similar level of accuracy. A parallel algorithmic approach driven by the knowledge of the internal microprocessor architecture allows significant speedup to be obtained while preserving standard sensitivity for similarity search problems.

Cathedral: Total Centralized Control Design, implement, test, release • Bazaar: Total Decentralization Release early, release often, make users partners in software development "Given enough eyeballs, all bugs are shallow" Code errors, logical errors, and architectural errors A combination of the two seems more sensible UPK,SB,PR GRC, IIT Bombay GCC-Par: GCC ≡ The Great Compiler Challenge 13/147 Another Example of The Generation Related Gap Locating the main function in the directory gcc-4.4.2/gcc using cscope File Line 0 collect2.c 766 main (int argc, char **argv) 1 fix-header.c 1074 main (int argc, char **argv) 2 fp-test.c 85 main (void ) 3 gcc.c 6216 main (int argc, char **argv) 4 gcov-dump.c 76 main (int argc ATTRIBUTE_UNUSED, char **argv 5 gcov-iov.c 29 main (int argc, char **argv) 6 gcov.c 355 main (int argc, char **argv) 7 gen-protos.c 130 main (int argc ATTRIBUTE_UNUSED, char **argv 8 genattr.c 89 main (int argc, char **argv) 9 genattrtab.c 4438 main (int argc, char **argv) a genautomata.c 9321 main (int argc, char **argv) b genchecksum.c 65 main (int argc, char ** argv) c gencodes.c 51 main (int argc, char **argv) d genconditions.c 209 main (int argc, char **argv) e genconfig.c 261 main (int argc, char **argv) f genconstants.c 50 main (int argc, char **argv) UPK,SB,PR GRC, IIT Bombay PPoPP'10 GCC-Par: GCC ≡ The Great Compiler Challenge 13/147 Another Example of The Generation Related Gap Locating the main function in the directory gcc-4.4.2/gcc using cscope File Line 0 collect2.c 766 main (int argc, char **argv) 1 fix-header.c 1074 main (int argc, char **argv) 2 fp-test.c 85 main (void ) 3 gcc.c 6216 main (int argc, char **argv) 4 gcov-dump.c 76 main (int argc ATTRIBUTE_UNUSED, char **argv 5 gcov-iov.c 29 main (int argc, char **argv) 6 gcov.c 355 main (int argc, char **argv) 7 gen-protos.c 130 main (int argc ATTRIBUTE_UNUSED, char **argv 8 genattr.c 89 main (int argc, char **argv) 9 genattrtab.c 4438 main (int argc, char **argv) a genautomata.c 9321 main (int argc, char **argv) b genchecksum.c 65 main (int argc, char ** argv) c gencodes.c 51 main (int argc, char **argv) d genconditions.c 209 main (int argc, char **argv) e genconfig.c 261 main (int argc, char **argv) f genconstants.c 50 main (int argc, char **argv) UPK,SB,PR GRC, IIT Bombay PPoPP'10 GCC-Par: GCC ≡ The Great Compiler Challenge 13/147 Another Example of The Generation Related Gap Locating the main function in the directory gcc-4.4.2/gcc using cscope g genemit.c 820 main (int argc, char **argv) h genextract.c 394 main (int argc, char **argv) i genflags.c 231 main (int argc, char **argv) j gengenrtl.c 350 main (int argc, char **argv) k gengtype.c 3584 main (int argc, char **argv) l genmddeps.c 45 main (int argc, char **argv) m genmodes.c 1376 main (int argc, char **argv) n genopinit.c 472 main (int argc, char **argv) o genoutput.c 1005 main (int argc, char **argv) p genpeep.c 353 main (int argc, char **argv) q genpreds.c 1399 main (int argc, char **argv) r genrecog.c 2718 main (int argc, char **argv) s main.c 33 main (int argc, char **argv) t mips-tdump.c 1393 main (int argc, char **argv) u mips-tfile.c 655 main (void ) v mips-tfile.c 4690 main (int argc, char **argv) w protoize.c 4373 main (int argc, char **const argv)

Modern Molecular Dynamics methods are employed to study quantum manybody systems, chemically reactive systems including explicit electronic degrees of freedom, and combinations thereof, as well as large classical biomolecular systems. Thus, complex problems such as isotope effects on enzymatic reactions can now be examined, routinely. In this article, modern molecular dynamics methods are reviewed and their application to quantum manybody systems and electronic structure calculations described. The resulting methodology, however, while powerful, is computationally intensive. Therefore, the mathematical structure of the techniques has been exploited to develop distributed memory parallel algorithms employing multiple levels of discretization. These multilevel-parallel methods are efficient and permit the large complex systems, such as enzyme catalysis, to be treated easily. In addition, it is shown how modern object oriented programming paradigms can be employed to implement multilevel parallel algorithms in a large computational package rapidly and efficiently. Finally, results and timings obtaining using the PINY_MD package developed by the authors are given for a variety of novel systems.

Modern Molecular Dynamics methods are employed to study quantum manybody systems, chemically reactive systems including explicit electronic degrees of freedom, and combinations thereof, as well as large classical biomolecular systems. Thus, complex problems such as isotope effects on enzymatic reactions can now be examined, routinely. In this article, modern molecular dynamics methods are reviewed and their application to quantum manybody systems and electronic structure calculations described. The resulting methodology, however, while powerful, is computationally intensive. Therefore, the mathematical structure of the techniques has been exploited to develop distributed memory parallel algorithms employing multiple levels of discretization. These multilevel-parallel methods are efficient and permit the large complex systems, such as enzyme catalysis, to be treated easily. In addition, it is shown how modern object oriented programming paradigms can be employed to implement multilevel parallel algorithms in a large computational package rapidly and efficiently. Finally, results and timings obtaining using the PINY_MD package developed by the authors are given for a variety of novel systems.

In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences. This could herald a new era of scientific exploration, bringing significant advancements across sectors from drug development to renewable energy. To answer this call, we present DeepSpeed4Science initiative (deepspeed4science.ai) which aims to build unique capabilities through AI system technology innovations to help domain experts to unlock today's biggest science mysteries. By leveraging DeepSpeed's current technology pillars (training, inference and compression) as base technology enablers, DeepSpeed4Science will create a new set of AI system technologies tailored for accelerating scientific discoveries by addressing their unique complexity beyond the common technical approaches used for accelerating generic large language models (LLMs). In this paper, we showcase the early progress we made with DeepSpeed4Science in addressing two of the critical system challenges in structural biology research.

K-means clustering, a fundamental unsupervised machine learning technique, is widely used in anomaly detection, image recognition, and customer segmentation. Traditional Python implementations, especially those using NumPy, face performance challenges with large, high-dimensional datasets due to Python's interpreted nature and dynamic typing. This paper introduces an innovative approach using the Mojo programming language, designed for AI development, to significantly improve the performance of the k-means clustering. Mojo combines Python's usability with the performance of system programming languages by offering features like vectorization, parallelization, and strong typing. We compare a NumPy-based Python implementation with an optimized Mojo implementation, detailing the translation process and optimization techniques, including Mojo's support for Single Instruction, Multiple Data (SIMD) operations, explicit memory management, and efficient data structures. These features significantly accelerate distance calculations crucial to the k-means algorithm. Benchmarks on synthetic datasets with varying sample sizes, feature counts, and cluster numbers demonstrate that the Mojo implementation consistently outperforms both the standard Python implementation and the highly optimized sci-kit-learn k-means, achieving speedups of 6x to 250x. These results highlight Mojo's potential as a powerful tool for high-performance data analysis, particularly for computationally demanding algorithms like k-means clustering, and contribute to high-performance computing in machine learning. This research sets the stage for further exploration of Mojo's applicability to other algorithms and hardware-specific optimizations for modern computing architectures.

Pipeline is a fundamental parallel programming pattern. Mainstream pipeline programming frameworks count on data abstractions to perform pipeline scheduling. This design is convenient for data-centric pipeline applications but inefficient for algorithms that only exploit task parallelism in pipeline. As a result, we introduce a new task-parallel pipeline programming framework called Pipeflow. Pipeflow does not design yet another data abstraction but focuses on the pipeline scheduling itself, enabling more efficient implementation of task-parallel pipeline algorithms than existing frameworks. We have evaluated Pipeflow on both micro-benchmarks and real-world applications. As an example, Pipeflow outperforms oneTBB 24% and 10% faster in a VLSI placement and a timing analysis workloads that adopt pipeline parallelism to speed up runtimes, respectively.

In the future, if we are to continue to expect improved application performance we will have to achieve it by exploiting course-grained hardware parallelism rather then simply relying on processor cycles getting faster. Programmers will, therefore, need to accept some of the burden of detecting and exploiting application level parallelism because automatic parallelization is still far from a reality. On

this paper we present COLT HPF (COordination Layer for Tasks expressed in HPF), a portable coordination /communication layer for HPF tasks. COLT HPF is implemented on top of PVM and provides suitable mechanisms for starting, even at run{time, instances of data-parallel tasks on disjoint groups of (PVM virtual) processors, along with optimized primitives for intertask communication where data to be exchanged may be distributed among the processors according to userspeci ed HPF directives.

Experience in applicative fields, above all deriving from the development of multidisciplinary parallel applications, seems to suggest a model where art outer coordination level is provided to allow data parallel tasks to run concurrently and to cooperate each other. The inner computational level of this coordination model can easily be expressed with HPF, a high-level data-parallel language. According to this model, we devised COLT, s, a coordination architectural layer that supports dynamic creation and concurrent execution of HPF tasks, and permits these tasks to cooperate though message passing. This paper proposes the exploitation of COLT~F by means of a simple skeleton-based coordination language and the associated source-to-source compiler. Differently from other related proposals, COLT~pF is portable and can exploit commercial, standard-compliant, HPF compilation systems. We used a physics application as a test-case for our approach, and we present the results of several experiments conducted on a cluster of Linux SMPs.

Model parallelism is a standard paradigm to decouple a deep neural network (DNN) into sub-nets when the model is large. Recent advances in class parallelism significantly reduce the communication overhead of model parallelism to a single floating-point number per iteration. However, traditional faulttolerance schemes, when applied to class parallelism, require storing the entire model on the hard disk. Thus, these schemes are not suitable for soft and frequent system noise such as stragglers (temporarily slow worker machines). In this paper, we propose an erasure-coding based redundant computing technique called robust class parallelism to improve the error resilience of model parallelism. We show that by introducing slight overhead in the computation at each machine, we can obtain robustness to soft system noise while maintaining the low communication overhead in class parallelism. More importantly, we show that on standard classification tasks, robust class parallelism maintains the stateof-the-art performance.

Nearly two decades of research in the area of Inductive Logic Programming (ILP) have seen steady progress in clarifying its theoretical foundations and regular demonstrations of its applicability to complex problems in very diverse domains. These results are necessary, but not sufficient, for ILP to be adopted as a tool for data analysis in an era of very large machine-generated scientific and industrial datasets, accompanied by programs that provide ready access to complex relational information in machine-readable forms (ontologies, parsers, and so on). Besides the usual issues about the ease of use, ILP is now confronted with questions of implementation. We are concerned here with two of these, namely: can an ILP system construct models efficiently when (a) Dataset sizes are too large to fit in the memory of a single machine; and (b) Search space sizes becomes prohibitively large to explore using a single machine. In this paper, we examine the applicability to ILP of a popular distributed computing approach that provides a uniform way for performing data and task parallel computations in ILP. The MapReduce programming model allows, in principle, very large numbers of processors to be used without any special understanding of the underlying hardware or software involved. Specifically, we show how the MapReduce approach can be used to perform the coverage-test that is at the heart of many ILP systems, and to perform multiple searches required by a greedy set-covering algorithm used by some popular ILP systems. Our principal findings with synthetic and real-world datasets for both data and task parallelism are these: (a) Ignoring overheads, the time to perform the computations concurrently increases with the size of the dataset for data parallelism and with the size of the search space for task parallelism. For data parallelism this increase is roughly in proportion to increases in dataset size; (b) If a MapReduce implementation is used as part of an ILP system, then benefits for data parallelism can only be expected above some minimal

The main objective of compiler and processor designers is to effectively exploit the instruction-level parallelism (ILP) available in applications. Although most of the times their research activities have been conducted separately, we believe that a stronger co-operation between them will make effective the increase of potential ILP coming from future architectures. Nowadays, most computer architecture achievements proceed towards the overcoming of the hurdle imposed by dependencies in the code, by means of extracting parallelism from large instruction windows. However, implementation constraints limit the size of this window and therefore the visibility of the program structure at run-time. In this paper we show the existence of distant parallelism that future compilers could detect. By distant parallelism we mean parallelism that can not be captured by the processor instruction window and that can produce threads suitable for parallel execution in a multithreaded processor. Although this parallelism also exists in numerical applications (going far beyond classical loop parallelism and usually known as task parallelism), we focus on non-numerical applications, where the data and computation structures make difficult the detection of concurrent threads of execution. Some preliminary but encouraging results are presented in the paper, reporting speed-ups in the range of 1.2 to 2.65. These results seem promising and want to show a new insight in the detection of threads for current and future multithreaded architectures. It is important to notice at this point that the benefits described herein are totally orthogonal to any other architectural techniques targeting a single thread.

Complex objects to support non-standard database applications require the use of substantial computing resources because their powerful operations must be performed and maintained in an interactive environment. Since the exploitation of parallelism within such operations seems to be promising, we investigate the principal approaches for processing a query on complex objects (molecules) in parallel. A number of arguments favor methods based on inter-molecule parallelism as against intra-molecule parallelism. Retrieval of molecules may be optimized by multiple storage structures and access paths. Hence, maintenance of such storage redundancy seems to be another good application area to explore the use of parallelism.

I must give a thank to libre software. It has provided me with a lot of material to study and learn from, play with, contribute to. I can't remember something I took at look at, as fortuitous as it may have seemed at the time, that didn't end up being useful at some point for my work, one way or the other. This common good is there to stay, for everybody's benefit, the software heritage slogan is really not just a catchword. Un grand merci aussi aux différents orchestres et groupes dans lesquels j'ai pu jouer de très divers styles de musique sur divers instruments, histoire de faire autre chose que de taper sur un clavier (d'ordinateur). Une mention spéciale à Manu qui ne s'est jamais lassé de corriger mes défauts au trombone. Un petit mot aussi pour mes différents amis, ici et là, que j'arrive à voir plus ou moins souvent, parfois cela se compte en années, mais c'est toujours avec autant de plaisir. Et enfin merci à ma famille, qui m'a offert le cadre qui m'a lancé jusqu'ici, en m'apprenant la curiosité, la ténacité, la générosité, ... x Contents Chapter 2 Using a task-based programming model for HPC Contents 2.1 The revival of a well-known paradigm .

This work proposes RaNNC (Rapid Neural Network Connector) as middleware for automatic hybrid parallelism. In recent deep learning research, as exemplified by T5 and GPT-3, the size of neural network models continues to grow. Since such models do not fit into the memory of accelerator devices, they need to be partitioned by model parallelism techniques. Moreover, to accelerate training for huge training data, we need a combination of model and data parallelisms, i.e., hybrid parallelism. Given a model description for PyTorch without any specification for model parallelism, RaNNC automatically partitions the model into a set of subcomponents so that (1) each subcomponent fits a device memory and (2) a high training throughput for pipeline parallelism is achieved by balancing the computation times of the subcomponents. Since the search space for partitioning models can be extremely large, RaNNC partitions a model through the following three phases. First, it identifies atomic subcomponents using simple heuristic rules. Next it groups them into coarser-grained blocks while balancing their computation times. Finally, it uses a novel dynamic programming-based algorithm to efficiently search for combinations of blocks to determine the final partitions. In our experiments, we compared RaNNC with two popular frameworks, Megatron-LM (hybrid parallelism) and GPipe (originally proposed for model parallelism, but a version allowing hybrid parallelism also exists), for training models with increasingly greater numbers of parameters. In the pretraining of enlarged BERT models, RaNNC successfully trained models five times larger than those Megatron-LM could, and RaNNC's training throughputs were comparable to Megatron-LM's when pre-training the same models. RaNNC also achieved better training throughputs than GPipe on both the enlarged BERT model pre-training (GPipe with hybrid parallelism) and the enlarged ResNet models (GPipe with model parallelism) in all of the settings we tried. These results are remarkable, since RaNNC automatically partitions models without any modification to their descriptions; Megatron-LM and GPipe require users to manually rewrite the models' descriptions.

Many institutions (e.g., universities, banks) are moving towards clusters. These clusters consist of commodity components such as PCs connected by fast networks. However, the single factor limiting the harnessing of the enormous computing power of these clusters for parallel computing is the lack of appropriate software. Present operating systems are not built to support parallel computing -they do not provide services to manage parallelism. Managing the available parallelism in a cluster means managing parallel processes and computational resources, in order to achieve high performance and use computational resources efficiently, and to make programming and use of the parallel system easy. Parallelism management in parallel programming tools, distributed shared memory and enhanced operating system environments has been in the majority of trials left to application programmers. Programmers must deal not only with programming of communication and coordination of parallel processes to achieve the correct execution of an application, but also with the problems of initialisation and control of the execution on the cluster. Users do not see a cluster as a single powerful computer. Furthermore, parallel systems are seen as being user unfriendly, due to their complexity. An analysis of the existing clusters shows that parallelism management systems are being developed and offered using two different approaches: middleware, at the application level; and underware, at the kernel level. Both of them have advantages and disadvantages. The problem is how to take advantage of both of them to create the best solution from the point of view of the execution performance, programmers and efficient use of resources. In the majority of execution environments programmers of parallel application cannot make a choice between the message passing and distributed shared memory communication paradigm. Both paradigms have advantages and disadvantages. The former is fast but difficult to use, the latter is easy to use but demonstrates reduced performance. These communication paradigms and the systems supporting them are treated independently of an operating system, rather than to be parts of a comprehensive operating system as they manage system resources. Parallel processing can be divided into the initialisation, execution and termination phases. Currently, researchers and manufacturers mainly concentrate their work on the execution phase in order to achieve the best performance. Ease of use of parallel systems and programmer's time are neglected. This approach discourages application programmers from parallel processing, as they have to program many activities, which are of an operating system nature, in particular those of the initialization and termination phases. These activities should be carried out auto matically by a cluster operating system to relieve programmers from error prone and time-consuming activities. In this talk an overview of our work carried out toward the devel opment of cluster operating systems that automatically and dynamically support parallel processing is presented. There are a number of aims of this talk. The first aim is to identify and discuss the basic issues of and solutions to the problem of the management of parallel processing on clusters. The second aim is to propose a new class of cluster operating systems that provide these services. In particular, these operating systems should: guarantee high performance of parallel processing on clusters and the efficient use of resources; support execution on a cluster of both message passing and shared memory based parallel applications; relieve programmers from error prone and time consuming work of allocation of processes to computers, management of interprocess communication and process synchronisation; provide transparency; and make the whole cluster based parallel system easy to use. The third aim is to introduce and discuss the architecture and services of a cluster operating system, called GENESIS, that allow the above specified goals to be achieved. The fourth

A number of computational applications lack instruction-level parallelism. This loss is particularly acute on sequences of dependent instructions on wide-issue or deeply pipelined architectures. We consider four real applications from computational biology, cryptanalysis, and data compression. These applications are characterized by long sequences of dependent instructions, irregular control-flow and intricate scalar and memory dependence patterns. While these benchmarks exhibit good memory locality and branch-predictability, state-of-the-art compiler optimizations fail to exploit much instruction-level parallelism. This paper shows that major performance gains are possible on such applications, through a loop transformation called deep jam. This transformation reshapes the control-flow of a program to facilitate the extraction of independent computations through classical back-end techniques. Deep jam combines accurate dependence analysis and control speculation, with a generalized form of recursive, multi-variant unroll-and-jam; it brings together independent instructions across irregular control structures, removing memory-based dependences through scalar and array renaming. This optimization contributes to the extraction of fine-grain parallelism in irregular applications. We propose a feedback-directed deep jam algorithm, selecting a jamming strategy, function of the architecture and application characteristics.

The main objective of compiler and processor designers is to effectively exploit the instruction-level parallelism (ILP) available in applications. Although most of the times their research activities have been conducted separately, we believe that a stronger cooperation between them will make effective the increase of potential ILP coming from future architectures. Nowadays, most computer architecture achievements proceed towards the overcoming of the hurdle imposed by dependencies in the code, by means of extracting parallelism from large instruction windows. However, implementation constraints limit the size of this window and therefore the visibility of the program structure at run-time. In this paper we show the existence of distant parallelism that future compilers could detect. By distant parallelism we mean parallelism that can not be captured by the processor instruction window and that can produce threads suitable for parallel execution in a multithreaded processor. Although this parallelism also exists in numerical applications (going far beyond classical loop parallelism and usually known as task parallelism), we focus on non-numerical applications, where the data and computation structures make difficult the detection of concurrent threads of execution. Some preliminary but encouraging results are presented in the paper, reporting speed-ups in the range of 1.2 to 2.65. These results seem promising and want to show a new insight in the detection of threads for current and future multithreaded architectures. It is important to notice at this point that the benefits described herein are totally orthogonal to any other architectural techniques targeting a single thread.

Task-based libraries such as Intel's Threading Building Blocks (TBB) provide higher levels of abstraction than threads for parallel programming. Work remains, however, to determine how straightforward it is to use these libraries to express various patterns of parallelism. This case study focuses on a particular pattern: pipeline parallelism. We attempted to transform three representative applicationscontent-based image retrieval, compression and video encoding-to pipelines using TBB. We successfully converted two of the three applications. In the successful cases we discuss our transformation process and contrast the expressivity and performance of our implementations to existing Pthreads versions; in the unsuccessful case, we detail what the challenges were and propose potential solutions.

In this paper we present ALDIMS, a language that combines the expressibility of general functional (MIMD) parallelism with compact expressibility of data (SPMD) parallelism. It uses distributed data structures for specifying data partitions and single assignment variables as abstract means of inter-process communication. Constructs for unstructured parallelism and process placement specifications make general MIMD parallelism expressible. We describe the issues of implementing process invocation and communication primitive generation. We also discuss source level parallelization and optimization issues and strategies.

H.264/AVC has been introduced in recent years to decrease the bit-rate and to increase the flexibility of implementations. After careful study and analysis, we have concluded that the complexity of this video codec depends mainly on its multidimensional data dependency, its elementary processing modules and its various profiles and levels. In this paper, we have proposed several Array-OL models especially for the modelling of data flow between the processing modules for self-generation of vhdl code of a memory controller for H.264/AVC. The controller will be adapted to the application profiles and levels and the used external memory. The proposed models combined with high level modelling tools should be used to perform embedded systems; the goal is the automatic generation of the Netlist from a high level description. The methodology is demonstrated by an example in which a specific level of the H.264/AVC is generated from information given by the proposed Array-OL models. The generated vhdl code is synthesised using two FPGA development board with ratios of used LUTs that do not exceed 10% and verified to work at 263 MHz frequency.

We study a class of deep neural networks with architectures that form a directed acyclic graph (DAG). For backpropagation defined by gradient descent with adaptive momentum, we show weights converge for a large class of nonlinear activation functions. The proof generalizes the results of Wu et al. (2008) who showed convergence for a feed-forward network with one hidden layer. For an example of the effectiveness of DAG architectures, we describe an example of compression through an AutoEncoder, and compare against sequential feed-forward networks under several metrics.

Applications containing compute-intensive kernels with nested loops can effectively leverage FPGAs to exploit fineand coarse-grained parallelism. HLS tools used to translate these kernels from high-level languages (e.g., C/C++), however, are inefficient in exploiting multiple levels of parallelism automatically, thereby producing sub-optimal accelerators. Moreover, the large design space resulting from the various combinations of fineand coarse-grained parallelism options makes exhaustive design space exploration prohibitively time-consuming with HLS tools. Hence, we propose a rapid estimation framework, MPSeeker, to evaluate performance/area metrics of various accelerator options for an application at an early design phase. Experimental results show that MPSeeker can rapidly (in minutes) explore the complex design space and accurately estimate performance/area of various design points to identify the near-optimal (95.7% performance of the optimal on average) combination of parallelism options.

Le paradigme de la programmation chimique a ete introduit a la fin des annees 1980 comme une maniere elegante de definir mathematiquement des programmes repartis. Le principe repose sur l’analogie des reactions chimiques, dans lequel un ensemble de molecules reagissent pour en former de nouvelles. La programmation chimique consiste a declarer des donnees initiales typees ainsi que des operateurs. L’ordre des reactions - la maniere dont sont appliques les operateurs sur les donnees - est resolu a l’execution, sans indications de la part du developpeur. Les programmes chimiques sont par nature paralleles et non deterministes. Ce paradigme a ete utilise pour les grappes et grilles de calculateurs et il reste pertinent pour les processeurs many-coeurs. Une grande part de la complexite des programmes chimiques reside dans le logiciel systeme qui a la charge d’orchestrer les reactions. L’implementation de ce logiciel fait resurgir les problemes classiques d’execution en environnement repa...

Two ways to exploit chips with a very large number of transistors are multicore processors and programmable logic chips. Some data parallel algorithms can be executed efficiently on ordinary parallel computers, including multicores. A class of data parallel algorithms is identified which have characteristics that make implementation on multiprocessors inefficient, but they are well suited for direct design as digital circuits. This leads to a programming model called circuit parallelism. The characteristics of circuit parallel algorithms are discussed, and a prototype system for supporting them is described.

A number of computational applications lack instruction-level parallelism. This loss is particularly acute on sequences of dependent instructions on wide-issue or deeply pipelined architectures. We consider four real applications from computational biology, cryptanalysis, and data compression. These applications are characterized by long sequences of dependent instructions, irregular control-flow and intricate scalar and memory dependence patterns. While these benchmarks exhibit good memory locality and branch-predictability, state-of-the-art compiler optimizations fail to exploit much instruction-level parallelism. This paper shows that major performance gains are possible on such applications, through a loop transformation called deep jam. This transformation reshapes the control-flow of a program to facilitate the extraction of independent computations through classical back-end techniques. Deep jam combines accurate dependence analysis and control speculation, with a generalized form of recursive, multi-variant unroll-and-jam; it brings together independent instructions across irregular control structures, removing memory-based dependences through scalar and array renaming. This optimization contributes to the extraction of fine-grain parallelism in irregular applications. We propose a feedback-directed deep jam algorithm, selecting a jamming strategy, function of the architecture and application characteristics. * . While at LRC Itaca, University of Versailles and CEA DAM. 1. Deep jam was first presented in a conference . This longer version describes a compilation algorithm at much greater depth, and draws a pragmatic roadmap to implement deep jam while optimizing the profitability of its steering heuristics. It also reports on the successful optimization of two additional applications, including the SPEC benchmark Gzip.

We introduce a new balanced assignment of experts (BASE) layer for large language models that greatly simplifies existing high capacity sparse layers. Sparse layers can dramatically improve the efficiency of training and inference by routing each token to specialized expert modules that contain only a small fraction of the model parameters. However, it can be difficult to learn balanced routing functions that make full use of the available experts; existing approaches typically use routing heuristics or auxiliary expert-balancing loss functions. In contrast, we formulate token-to-expert allocation as a linear assignment problem, allowing an optimal assignment in which each expert receives an equal number of tokens. This optimal assignment scheme improves efficiency by guaranteeing balanced compute loads, and also simplifies training by not requiring any new hyperparameters or auxiliary losses. Code is publicly released.

The massive addition of cores on a chip is adding more pressure to the accesses to main memory. In order to avoid this bottleneck, we propose the use of a simple producer-consumer model, which allows for the temporary results to be transferred directly from one task to another. These data transfer operations are performed within the chip, using on-chip memory, thus avoiding costly off-chip memory accesses. We implement this model on a real many-core processor, the 48-core Intel Single-chip Cloud Computer processor using its on-chip memory facilities. We find that the Producer-Consumer model adapts to such architectures and allow to achieve good task and data parallelism. For the evaluation of the proposed platform we implement a graph-based application using the Producer-Consumer model. Our tests show that the model scales very well as it takes advantage of the on-chip memory. The execution times of our implementation are up to 9 times faster than the baseline implementation, which relies on storing the temporary results to main memory.

This paper addresses the design of visual paradigms for observing the parallel execution of logic programs. First, an intuitive method is proposed for arriving at the design of a paradigm and its implementation as a tool for a given model of parallelism. This method is based on stepwise refinement starting from the definition of basic notions such as events and observables and some precedence relationships among events which hold for the given model of parallelism. The method is then applied to several types of parallel execution models for logic programs (Orparallelism, Determinate Dependent And parallelism, Restricted And-parallelism) for which visualization paradigms are designed. Finally, VisAndOr, a tool which implements all of these paradigms is presented, together with a discussion of its usefulness through examples.