Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Related Work
Checkpoint Based Methods
Other Methods
Theoretical Foundation of the Mpich-V Protocol
Definition 2 (Pessimistic-Log Protocols) Let
Conclusion and Future Work
References

MPICH-V: Toward a Scalable Fault Tolerant MPI for Volatile Nodes

Profile image of O. LodygenskyO. LodygenskyProfile image of George BosilcaGeorge Bosilca

2002, ACM/IEEE SC 2002 Conference (SC'02)

https://doi.org/10.1109/SC.2002.10048
visibility

…

description

18 pages

link

1 file

Abstract

Global Computing platforms, large scale clusters and future TeraGRID systems gather thousands of nodes for computing parallel scientific applications. At this scale, node failures or disconnections are frequent events. This Volatility reduces the MTBF of the whole system in the range of hours or minutes. We present MPICH-V, an automatic Volatility tolerant MPI environment based on uncoordinated checkpoint/ rollback and distributed message logging. MPICH-V architecture relies on Channel Memories, Checkpoint servers and theoretically proven protocols to execute existing or new, SPMD and Master-Worker MPI applications on volatile nodes. To evaluate its capabilities, we run MPICH-V within a framework for which the number of nodes, Channels Memories and Checkpoint Servers can be completely configured as well as the node Volatility. We present a detailed performance evaluation of every component of MPICH-V and its global performance for non-trivial parallel applications. Experimental results demonstrate good scalability and high tolerance to node volatility. 0-7695-1524-X/02 $17.00 (C) 2002 IEEE

Related papers

SHIELD: a fault-tolerant MPI for an infiniband cluster
Shin Kim

2006

Today's high performance cluster computing technologies demand extreme robustness against unexpected failures to finish aggressively parallelized work in a given time constraint. Although there has been a steady effort in developing hardware and software tools to increase fault-resilience of cluster environments, a successful solution has yet to be delivered to commercial vendors. This paper presents SHIELD, a practical and easily-deployable fault-tolerant MPI and management system of MPI for an Infiniband cluster. SHIELD provides a novel framework that can be easily used in real cluster systems, and it has different design perspectives than those proposed by other fault-tolerant MPI. We show that SHIELD provides robust fault-resilience to fault-vulnerable cluster systems and that the design features of SHIELD are useful wherever fault-resilience is regarded as the matter of utmost importance.

View PDFchevron_right
Improved message logging versus improved coordinated checkpointing for fault tolerant MPI
Geraud Krawezik

2004 IEEE International Conference on Cluster Computing (IEEE Cat. No.04EX935)

Fault tolerance is a very important concern for critical high performance applications using the MPI library. Several protocols provide automatic and transparent fault detection and recovery for message passing systems with different impact on application performance and the capacity to tolerate a high fault rate. In a recent paper, we have demonstrated that the main differences between pessimistic sender based message logging and coordinated checkpointing are 1) the communication latency and 2) the performance penalty in case of faults. Pessimistic message logging increases the latency, due to additional blocking control messages. When faults occur at a high rate, coordinated checkpointing implies a higher performance penalty than message logging due to a higher stress on the checkpoint server. In this paper we extend this study to improved versions of message logging and coordinated checkpoint protocols which respectively reduces the latency overhead of pessimistic message logging and the server stress of coordinated checkpoint. We detail the protocols and their implementation into the new MPICH-V fault tolerant framework. We compare their performance against the previous versions and we compare the novel message logging protocols against the improved coordinated checkpointing one using the NAS benchmark on a typical high performance cluster equipped with a high speed network. The contribution of this paper is two folds: a) an original message logging protocol and an improved coordinated checkpointing protocol and b) the comparison between them.

View PDFchevron_right
LA-MPI : The Design and Implementation of a Network-Fault-Tolerant MPI for Terascale Clusters
Nehal Desai

In this paper we discuss unique architectural elements of the Los Alamos Message Passing Interface (LA-MPI). LAMPI is a high-performance, network fault-tolerant, threadsafe MPI library designed for terascale clusters that are inherently unreliable due to their sheer number of system components and inherent trade-offs between cost and performance. We examine in detail the design concepts used to implement LA-MPI. These include reliability features, such as application-level checksumming, message retransmission, and automatic message re-routing. Other key performance enhancing features, such as concurrent message routing over multiple, diverse network adapters and protocols, and communication-specific optimizations (e.g., shared memory) are examined. Email: lampi-support@lanl.gov. Los Alamos report LAUR-03-0939. Los Alamos National Laboratory is operated by the University of California for the National Nuclear Security Administration of the United States Department of Energy under con...

View PDFchevron_right
Collective operations in application-level fault-tolerant MPI
daniel marques

2003

The running times of many computational science programs are now significantly greater than the mean-time-betweenfailures (MTBF) of the hardware they run on. Therefore, fault-tolerance is becoming a critical issue on highperformance platforms.

View PDFchevron_right
Checkpointing of Parallel MPI Applications Using MPI One-sided API with Support for Byte-addressable Non-volatile RAM
Maciej Maciejewski

Procedia Computer Science, 2016

The increasing size of computational clusters results in an increasing probability of failures, which in turn requires application checkpointing in order to survive those failures. Traditional checkpointing requires data to be copied from application memory into persistent storage medium, which increases application execution time as it is usually done in a separate step. In this paper we propose to use emerging byte-addressable non-volatile RAM (NVRAM) as a persistent storage medium and we analyze various methods of making consistent checkpoints with support of MPI one-sided API in order to minimize checkpointing overhead. We test our solution on two applications: HPCCG benchmark and PageRank algorithm. Our experiments showed that NVRAM based checkpointing performs much better than traditional disk based approach. We also simulated different possible latencies and bandwidth of future NVRAM and our experiments showed that only bandwidth had visible impact onto application execution time.

View PDFchevron_right
FT-MPI: Fault Tolerant MPI, Supporting Dynamic Applications in a Dynamic World
Aneli FT

2000

Initial versions of MPI were designed to work efficiently on multiprocessors which had very little job control and thus static process models, subsequently forcing them to support dynamic process operations would have effected their performance. As current HPC systems increase in size with higher potential levels of individual node failure, the need rises for new fault tolerant systems to be developed. Here we present a new implementation of MPI called FT-MPI 1 that allows the semantics and associated failure modes to be completely controlled by the application. Given is an overview of the FT-MPI semantics, design and some performance issues as well as the HARNESS g_hcore implementation it is built upon.

View PDFchevron_right
FEMPI: A Lightweight Fault-tolerant MPI for Embedded Cluster Systems
Adam Jacobs

… on Embedded Systems …, 2006

Ever-increasing demands of space missions for data returns from their limited processing and communications resources have made the traditional approach of data gathering, data compression, and data transmission no longer viable. Increasing on-board processing power by providing high-performance computing (HPC) capabilities using commercial-off-the-shelf (COTS) components is a promising approach that significantly increases performance while reducing cost. However, the susceptibility of COTS components to single-events upset (SEU) is a concern demanding fault-tolerant system infrastructure. Among the components of this infrastructure, message-passing middleware based upon the Message Passing Interface (MPI) standard is essential, so as to support and provide a nearly effortless transition for earth and space science applications in MPI from groundbased computational clusters to HPC systems in space. In this paper, we present the design of a fault-tolerant MPIcompatible middleware for embedded cluster computing known as FEMPI (Fault-tolerant Embedded MPI). We also present preliminary performance results with FEMPI on a COTS-based, embedded cluster system prototype.

View PDFchevron_right
Collective Operations in an Application-level Fault Tolerant MPI System
daniel marques

2003

The running times of many computational science programs are now significantly greater than the mean-time-betweenfailures (MTBF) of the hardware they run on. Therefore, fault-tolerance is becoming a critical issue on highperformance platforms.

View PDFchevron_right
Parallel Checkpoint/Restart for MPI Applications
Andrew Lumsdaine

As high-performance clusters continue to grow in size and popularity, issues of fault tolerance and reliability are becoming limiting factors on application scalability. To ad- dress these issues, we present the design and implementa- tion of a system for providing coordinated checkpointing and rollback recovery for MPI-based parallel applications. Our approach integrates the Berkeley Lab BLCR kernel- level process checkpoint system with the LAM implemen- tation of MPI through a defined checkpoint/restart inter- face. Checkpointing is transparent to the application, al- lowing the system to be used for cluster maintenance and scheduling reasons as well as for fault tolerance. Exper- imental results show negligible performance impact due to the incorporation of the checkpoint support capabilities into LAM/MPI.

View PDFchevron_right
Blocking vs. non-blocking coordinated checkpointing for large-scale fault tolerant MPI Protocols
Laurence Pilard

Future Generation …, 2008

A long-term trend in high-performance computing is the increasing number of nodes in parallel computing platforms, which entails a higher failure probability. Fault tolerant programming environments should be used to guarantee the safe execution of critical applications. Research in fault tolerant MPI has led to the development of several fault tolerant MPI environments. Different approaches are being proposed using a variety of fault tolerant message passing protocols based on coordinated checkpointing or message logging. The most popular approach is with coordinated checkpointing. In the literature, two different concepts of coordinated checkpointing have been proposed: blocking and non-blocking. However they have never been compared quantitatively and their respective scalability remains unknown. The contribution of this paper is to provide the first comparison between these two approaches and a study of their scalability. We have implemented the two approaches within the MPICH environments and evaluate their performance using the NAS parallel benchmarks.

View PDFchevron_right

References (23)

  1. A. Agbaria and R. Friedman. Starfish: Fault-tolerant dynamic MPI programs on clusters of workstations. In In 8th IEEE International Symposium on High Perfor- mance Distributed Computing, 1999.
  2. A.Selikhov, G.Bosilca, S.Germain, G.Fedak, and F.Cappello. MPICH-CM: a communication library design for P2P MPI implementation. In To appear in Proceedings of 9-th EuroPVM/MPI conference, Lecture Notes in Computer Science, Springer-Verlag, Berlin Heidelberg New York, 2002.
  3. D. Bailey, T. Harris, W. Saphir, R. van der Wijngaart, A. Woo, and M. Yarrow. The NAS Parallel Bench- marks 2.0. Report NAS-95-020, Numerical Aero- dynamic Simulation Facility, NASA Ames Research Center, Mail Stop T 27 A-1, Moffett Field, CA 94035- 1000, USA, December 1995.
  4. R. Batchu, J. Neelamegam, Z. Cui, M. Beddhua, A. Skjellum, Y. Dandass, and M. Apte. MPI/FT TM : Architecture and taxonomies for fault-tolerant, message-passing middleware for performance- portable parallel computing. In In Proceedings of the 1st IEEE International Symposium of Cluster Com- puting and the Grid held in Melbourne, Australia. , 2001.
  5. W. J. Bolosky, J. R. Douceur, D. Ely, and M. Theimer. Feasibility of a serverless distributed file system de- ployed on an existing set of desktop pcs. In Proceed- ings of the ACM international conference on Measure- ment and modeling of computer systems, SIGMET- RICS, pages 34-43, 2000.
  6. A. Brown and D. A. Patterson. Embracing fail- ure: A case for recovery-oriented computing (roc). In High Performance Transaction Processing Sympo- sium, Asilomar, CA, October 2001.
  7. Yuqun Chen, Kai Li, and James S. Plank. CLIP: A checkpointing tool for message-passing parallel pro- grams. In Proceedings the IEEE Supercomputing '97 Conference (SC97), november 1997.
  8. E. Elnozahy, D. Johnson, and Y. Wang. A survey of rollback-recovery protocols in message-passing sys- tems. In Technical Report CMU-CS-96-181, Carnegie Mellon University, October, 1996.
  9. E. N. Elnozahy and W. Zwaenepoel. Replicated dis- tributed processes in manetho. In FTCS-22: 22nd International Symposium on Fault Tolerant Comput- ing, pages 18-27, Boston, Massachusetts, 1992. IEEE Computer Society Press.
  10. G. Fagg and J. Dongarra. FT-MPI: Fault tolerant MPI, supporting dynamic applications in a dynamic world. In Euro PVM/MPI User's Group Meeting 2000 ,Springer-Verlag, Berlin, Germany, pp.346-353., 2000.
  11. G. Fagg and K. London. MPI interconnection and control. In Technical Report TR98-42, Major Shared Resource Center, U.S. Army Corps of Engineers Wa- terways Experiment Station, Vicksburg, Missippissi, 1998.
  12. I. Foster and N. Karonis. A grid-enabled MPI: Mes- sage passing in heterogeneous distributed computing systems. In In Proceedings of SC 98. IEEE, Nov., 1999.
  13. Ian T. Foster, Carl Kesselman, Gene Tsudik, and Steven Tuecke. A security architecture for computa- tional grids. In ACM Conference on Computer and Communications Security, pages 83-92, 1998.
  14. Edgar Gabriel, Michael Resch, and Roland Rhle. Im- plementing MPI with optimized algorithms for meta- computing. In Yoginder S. Dandass Anthony Skjel- lum, Purushotham V. Bangalore, editor, Proceedings of the Third MPI Developer's and User's Conference. MPI Software Technology Press, 1999.
  15. Cecile Germain, Vincent Neri, Gille Fedak, and Franck Cappello. XtremWeb: Building an experi- mental platform for global computing. In The 1st IEEE/ACM International Workshop on Grid Com- puing, Springer Verlag, LNCS 1971, pages 91-101, 2000.
  16. William Gropp, Ewing Lusk, Nathan Doss, and An- thony Skjellum. High-performance, portable imple- mentation of the MPI Message Passing Interface Stan- dard. Parallel Computing, 22(6):789-828, September 1996.
  17. D B. Johnson and W Zwaenepoel. Sender-based mes- sage logging. Technical report, Department of Com- puter Science, Rice University, Houston, Texas, 1987.
  18. T. T-Y. Juang and S. Venkatesan. Crash recovery with little overhead. In 11th Int. Conf on Distributed Com- puting Systems, ICDCS-11, pages 454-461, MAY 1991.
  19. Soulla Louca, Neophytos Neophytou, Arianos Lachanas, and Paraskevas Evripidou. MPI-FT: Portable fault tolerance scheme for MPI. In Parallel Processing Letters, Vol. 10, No. 4, 371-382 , World Scientific Publishing Company., 2000.
  20. P. N. Pruitt. An Asynchronous Checkpoint and Roll- back Facility for Distributed Computations. PhD the- sis, College of William and Mary in Virginia, May 1998.
  21. Sriram Rao, Lorenzo Alvisi, and Harrick M. Vin. Egida: An extensible toolkit for low-overhead fault- tolerance. In Symposium on Fault-Tolerant Comput- ing, pages 48-55, 1999.
  22. Georg Stellner. CoCheck: Checkpointing and Process Migration for MPI. In Proceedings of the 10th Inter- national Parallel Processing Symposium (IPPS '96), Honolulu, Hawaii, 1996.
  23. E. Strom and S. Yemini. Optimistic recovery in dis- tributed systems. In ACM Transactions on Computer Systems, volume 3(3), pages 204-226. ACM, Aug 1985.

Related papers

MPICH-V Project: A Multiprotocol Automatic Fault-Tolerant MPI
Geraud Krawezik

The International Journal of High Performance Computing Applications, 2006

High performance computing platforms such as Clusters, Grid and Desktop Grids are becoming larger and subject to more frequent failures. MPI is one of the most used message passing libraries in HPC applications. These two trends raise the need for fault-tolerant MPI. The MPICH-V project focuses on designing, implementing and comparing several automatic fault-tolerant protocols for MPI applications. We present an extensive related work section highlighting the originality of our approach and the proposed protocols. We then present four fault-tolerant protocols implemented in a new generic framework for fault-tolerant protocol comparison, covering a large spectrum of known approaches from coordinated checkpoint, to uncoordinated checkpoint associated with causal message logging. We measure the performance of these protocols on a micro-benchmark and compare them with the NAS benchmark, using an original fault tolerance test. Finally, we outline the lessons learned from this in depth fa...

View PDFchevron_right
Replication-Based Fault Tolerance for MPI Applications
Vipin Chaudhary

IEEE Transactions on Parallel and Distributed Systems, 2000

As computational clusters increase in size, their mean-time-to-failure reduces drastically. Typically, checkpointing is used to minimize the loss of computation. Most checkpointing techniques, however, require central storage for storing checkpoints. This results in a bottleneck and severely limits the scalability of checkpointing, while also proving to be too expensive for dedicated checkpointing networks and storage systems. We propose a scalable replication-based MPI checkpointing facility. Our reference implementation is based on LAM/MPI, however, it is directly applicable to any MPI implementation. We extend the existing state of fault-tolerant MPI with asynchronous replication, eliminating the need for central or network storage. We evaluate centralized storage, a Sun X4500-based solution, an EMC SAN, and the Ibrix commercial parallel file system and show that they are not scalable, particularly after 64 CPUs. We demonstrate the low overhead of our checkpointing and replication scheme with the NAS Parallel Benchmarks and the High Performance LINPACK benchmark with tests up to 256 nodes while demonstrating that checkpointing and replication can be achieved with much lower overhead than that provided by current techniques. Finally, we show that the monetary cost of our solution is as low as 25% of that of a typical SAN/parallel file system-equipped storage system.

View PDFchevron_right
A Scalable Asynchronous Replication-Based Strategy for Fault Tolerant MPI Applications
Vipin Chaudhary

Lecture Notes in Computer Science, 2007

As computational clusters increase in size, their mean-time-to-failure reduces. Typically checkpointing is used to minimize the loss of computation. Most checkpointing techniques, however, require a central storage for storing checkpoints. This severely limits the scalability of checkpointing. We propose a scalable replication-based MPI checkpointing facility that is based on LAM/MPI. We extend the existing state of fault-tolerant MPI with asynchronous replication, eliminating the need for central or network storage. We evaluate centralized storage, SAN-based solutions, and a commercial parallel file system, and show that they are not scalable, particularly beyond 64 CPUs. We demonstrate the low overhead of our replication scheme with the NAS Parallel Benchmarks and the High Performance LINPACK benchmark with tests up to 256 nodes while demonstrating that checkpointing and replication can be achieved with much lower overhead than that provided by current techniques.

View PDFchevron_right
Coordinated checkpoint versus message log for fault tolerant MPI
Geraud Krawezik

International Journal of High Performance Computing and Networking, 2004

MPI is one of the most adopted programming models for Large Clusters and Grid deployments. However, these systems often suffer from network or node failures. This raises the issue of selecting a fault tolerance approach for MPI. Automatic and transparent ones are based on either coordinated checkpointing or message logging associated with uncoordinated checkpoint. They are many protocols, implementations and optimizations for these approaches but few results about their comparison. Coordinated checkpoint has the advantage of a very low overhead on fault free executions. In contrary a message logging protocol systematically adds a significant message transfer penalty. The drawbacks of coordinated checkpoint come from its synchronization cost at checkpoint and restart times. In this paper we implement, evaluate and compare the two kinds of protocols with a special emphasis on their respective performance according to fault frequency. The main conclusion (under our experimental conditions) is that message logging becomes relevant for a large scale cluster from one fault every hour for applications with large dataset.

View PDFchevron_right
Blocking vs. Non-Blocking Coordinated Checkpointing for Large-Scale Fault Tolerant MPI
Laurence Pilard

Proceedings of the …, 2006

A long-term trend in high-performance computing is the increasing number of nodes in parallel computing platforms, which entails a higher failure probability. Fault tolerant programming environments should be used to guarantee the safe execution of critical applications. Research in fault tolerant MPI has led to the development of several fault tolerant MPI environments. Different approaches are being proposed using a variety of fault tolerant message passing protocols based on coordinated checkpointing or message logging. The most popular approach is with coordinated checkpointing. In the literature, two different concepts of coordinated checkpointing have been proposed: blocking and non-blocking. However they have never been compared quantitatively and their respective scalability remains unknown. The contribution of this paper is to provide the first comparison between these two approaches and a study of their scalability. We have implemented the two approaches within the MPICH environments and evaluate their performance using the NAS parallel benchmarks.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved