The cost of recovery in message logging protocols

2011

A look at Exascale reveals a future with multicore supercomputers that will inexorably experience frequent failures. Providing scalable and efficient fault tolerance support is one of the major concerns to pave the road for the next generation of machines. Checkpoint/restart remains as the standard de facto approach to provide fault tolerance in supercomputers. However, its high recovery cost has brought the attention of the community to an alternative mechanism, message logging. In this paper we present the design of a message logging protocol that targets multicore machines based on two fundamental assumptions. First, a multicore node is the minimum unit of failure and very frequently only one node goes down per failure. Second, the shared memory is a key resource to bring down the overhead of message logging. This paper also presents an analysis of failure data from recent supercomputers that show that most of the time a failure involves one single computational node. We offer two different distributions to model the data. Using those distributions we build a model for the survivability of the message logging protocol to multiple concurrent failures. We demonstrate our technique has a low overhead. The results of an experiment with a stencil program show the execution time penalty is below 5% when the program scales up to 1024 cores. Moreover, even when the protocol was designed to tolerate one single failure at a time, it provides a high probability of survival to a failure involving any number of nodes. Using real-world data from recent supercomputers we demonstrate the chances of survive any failure are higher than 99%.

Concurrency and Computation: Practice and Experience, 2009

In the rollback recovery of large-scale long-running applications in a distributed environment, pessimistic message logging protocols enable failed processes to recover independently, though at the expense of logging every message synchronously during fault-free execution. In contrast, coordinated checkpointing protocols avoid message logging, but they are poor in scalability with a sharply increased coordinating overhead as the system grows. With the aim of achieving efficient rollback recovery by trading off logging overhead and coordinating overhead, this paper suggests a partitioning of the system into clusters, and then presents a scheme to implement the conversion between these overheads. Using the proposed conversion, coordination can be introduced to reduce the unbearable logging overhead found in some systems, whereas proper logging can be employed to alleviate the unacceptable coordinating overhead in others. Furthermore, heuristics are introduced to address the issue of how to partition the system into clusters in order to speed up the recovery process and to improve recovery efficiency. Performance evaluation results indicate that our scheme can lower the overall system overhead effectively. Copyright

Using rollback-recovery based fault tolerance (FT) techniques in applications executed on Multicore Clusters is still a challenge, because the overheads added depend on the applications' behavior and resource utilization. Many FT mechanisms have been developed in recent years, but analysis is lacking concerning how parallel applications are affected when applying such mechanisms. In this work we address the combination of process mapping and FT tasks mapping on multicore environments. Our main goal is to determine the configuration of a pessimistic receiver-based message logging approach which generates the least disturbance to the parallel application. We propose to characterize the parallel application in combination with the message logging approach in order to determine the most significant aspects of the application such as computation-communication ratio and then, according to the values obtained, we suggest a configuration that can minimize the added overhead for each specific scenario. In this work we show that in some situations is better to save some resources for the FT tasks in order to lower the disturbance in parallel executions and also to save memory for these FT tasks. Initial results have demonstrated that when saving resources for the FT tasks we can achieve 25% overhead reduction when using a pessimistic message logging aproach as FT support.

Lecture Notes in Computer Science, 1996

A message is in-transit with respect to a global state if its sending is recorded in this global state, while its receipt is not. Checkpointing algorithms have to log such in-transit messages in order to restore the state of channels when a computation has to be resumed from a consistent global state after a failure has occurred. Coordinated checkpointing algorithms log those in-transit messages exactly on stable storage. Because of their lack of synchronization, uncoordinated checkpointing algorithms conservatively log more messages.

2009 IEEE International Conference on Cluster Computing and Workshops, 2009

With the growing scale of high performance computing platforms, fault tolerance has become a major issue. Among the various approaches for providing fault tolerance to MPI applications, message logging has been proved to tolerate higher failure rate. However, this advantage comes at the expense of a higher overhead on communications, due to latency intrusive logging of events to a stable storage. Previous work proposed and evaluated several protocols relaxing the synchronicity of event logging to moderate this overhead. Recently, the model of message logging has been refined to better match the reality of high performance network cards, where message receptions are decomposed in multiple interdependent events. According to this new model, deterministic and non-deterministic events are clearly discriminated, reducing the overhead induced by message logging. In this paper we compare, experimentally, a pessimistic and an optimistic message logging protocol, using this new model and implemented in the Open MPI library. Although pessimistic and optimistic message logging are, respectively, the most and less synchronous message logging paradigms, experiments show that most of the time their performance is comparable.

IEEE Transactions on Mobile Computing, 2002

Some extended algorithms of checkpointing coordination and pessimistic logging schemes have been proposed for the fault tolerant mobile systems. However, not much attention has been paid for the optimistic logging scheme. The optimistic logging requires the less failure-free operation cost compared to the other logging schemes and the less failure recovery cost compared to the checkpointing schemes. Hence, this paper presents an efficient scheme to implement optimistic message logging for the mobile computing environment. In the proposed scheme, to cope with the space problem of the mobile hosts and to utilize the volatile log space, the task of logging is assigned to the mobile support stations. Also, to reduce the message overhead, the mobile support stations also take care of the dependency tracking. The performance of the proposed scheme is evaluated. The simulation results confirm that the proposed scheme requires a little failure-free overhead. Also, the cost for the unnecessary recovery caused by the imprecise dependency can be adjustable in some extent, by properly selecting the logging frequency.

Twenty-Fifth International Symposium on Fault-Tolerant Computing. Digest of Papers, 1995

Much of the literature on message logging and checkpointing in the past decade has been based on a so-called optimistic approach 1] that places more emphasis on failure-free overhead than recovery eciency. Our experience has shown that most telecommunications systems use a pessimistic approach because the main purpose of using message logging and checkpointing is to achieve fast and localized recovery, and the failure-free overhead of a pessimistic approach can often be made reasonably low by exploiting application-speci c information.

High Performance Computing, 2017

With the exascale era within reach, the high performance computing community is preparing to embrace the challenges associated with extreme-scale systems. Resilience raises as one of the major hurdles in making those systems usable for the advance of science and industry. Message logging is a well-known strategy to provide fault tolerance, one that is promising due to its ability to avoid global restart. However, message-logging protocols may suffer considerable overhead if implemented for the general case. This paper introduces a new messagelogging protocol that leverages the benefits of a flexible parallel programming paradigm. We evaluate the protocol using a particular type of applications and demonstrate it can keep a low performance penalization when scaling up to 128,000 cores.

IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN 2012), 2012

Although many details of an eventual Exascale machine remain unknown, we can safely make a couple of assumptions. Exascale machines will be composed of multicore nodes and will experience frequent failures. The latter means that effective resilience support is imperative to make Exascale machines usable. The former opens up opportunities for exploring new alternatives to provide resilience support. This paper examines a new fault tolerance protocol for multicore systems. The paper contains three major parts. In the first part, we start by showing evidence that a node (and not a core) is the appropriate unit of failure. When a crash hits a machine, it usually renders unusable a whole node. Rarely, the crash brings down more than one node. The second part describes a message logging protocol that tolerates the failure of whole nodes and uses an efficient shared memory scheme to minimize overhead. We present results on various clusters and scale the approach to 1024 cores with a stencil computation. The overhead is always lower than 4%. The third part performs an analysis of reliability to understand how robust the protocol is when failures affect several nodes. Using an analytical framework and the frequency of multiple-node failures, we find that our approach is able to survive more than 99% of the crashes.

2004

This paper presents a causal message logging protocol with independent checkpointing for mobile nodes with the aim of efficiently handling several constraints of the mobile nodes such as mobility and disconnection, limited life of battery power, small amount of storage and low bandwidth on wireless link. For this purpose, the protocol includes a low-cost failure-free mechanism requiring only locating the mobility agent maintaining the latest checkpoint of each process on an mobile node during its handoff process. This mechanism forces only the latest checkpoint to be maintained on the stable storage while incurring low failure-free overhead. Also, the protocol uses two garbage collection schemes to remove log information of mobile nodes. The first scheme enables each mobile node to autonomously remove useless log information in its storage by piggybacking only some additional information without requiring any extra message and forced checkpoint. The second scheme allows the mobile node to remove a part of log information in its storage if more empty storage space is required after executing the first scheme. It reduces the number of processes to participate in the garbage collection by using the size of the log information of each process. Simulation results show that the two proposed schemes significantly reduce the garbage collection overhead compared with traditional schemes. Additionally, we present an efficient recovery algorithm to avoid frequent stable storage accesses, impose no restriction on the execution of live processes during recovery and ensure consistent recovery in case of being integrated with independent checkpointing.