Failure Recovery

The design and correctness of a communication facility for a distributed computer system are reported on. The facility provides support for fault-tolerant process groups in the form of a family of reliable multicast protocols that can be used in both local-and wide-area networks. These protocols attain high levels of concurrency, while respecting application-specific delivery ordering constraints, and have varying cost and performance that depend on the degree of ordering desired. In particular, a protocol that enforces causal delivery orderings is introduced and shown to be a valuable alternative to conventional asynchronous communication protocols. The facility also ensures that the processes belonging to a fault-tolerant process group will observe consistent orderings of events affecting the group as a whole, including process failures, recoveries, migration, and dynamic changes to group properties like member rankings. A review of several uses for the protocols in the ISIS system, which supports fault-tolerant resilient objects and bulletin boards, illustrates the significant simplification of higher level algorithms made possible by our approach.

A peer-to-peer technique called ZIGZAG for single-source media streaming is designed . ZIGZAG allows the media server to distribute content to many clients by organizing them into an appropriate tree rooted at the server. This application-layer multicast tree has a height logarithmic with the number of clients and a node degree bounded by a constant. This helps reduce the number of processing hops on the delivery path to a client while avoiding network bottleneck. Consequently, the end-to-end delay is kept small. Although one could build a tree satisfying such properties easily, an efficient control protocol between the nodes must be in place to maintain the tree under the effects of network dynamics and unpredictable client behaviors. ZIGZAG handles such situations gracefully requiring a constant amortized control overhead. Especially, failure recovery can be done regionally with little impact on the existing clients and mostly no burden on the server.

Availability is a storage system property that is both highly desired and yet minimally engineered. While many systems provide mechanisms to improve availability -such as redundancy and failure recovery -how to best configure these mechanisms is typically left to the system manager. Unfortunately, few individuals have the skills to properly manage the trade-offs involved, let alone the time to adapt these decisions to changing conditions. Instead, most systems are configured statically and with only a cursory understanding of how the configuration will impact overall performance or availability. While this issue can be problematic even for individual storage arrays, it becomes increasingly important as systems are distributed -and absolutely critical for the widearea peer-to-peer storage infrastructures being explored.

It is now well recognized that an effective service recovery program is essential to generating customer satisfaction and loyalty. A number of studies have investigated the impact of service recovery efforts (compensation, speed of response, etc) on post-recovery satisfaction. However, despite the importance of global markets, none have examined the impact of customers' cultural value orientation (i.e., cultural values measured at the individual level) in implementing effective service recovery programs. In this research we use the individual rather than the nation as the unit of analysis. Using an experimental design with data from both Eastern and Western cultures, we investigate how customer evaluations of recovery efforts are influenced by interplay of consumers' value orientation and service recovery attributes (apology, cognitive control, and recovery initiation). The results reveal that cultural values of individual Power Distance, Uncertainty Avoidance and Collectivism do indeed interact with a firm's recovery tactics to influence perceptions of fairness (justice). Finally, all three forms of justice (distributive, procedural, interactional) positively impact on overall service recovery satisfaction.

On May 17th 1999, NASA activated for the first time an AI-based planner/scheduler running on the flight processor of a spacecraft. This was part of the Remote Agent Experiment (RAX), a demonstration of closed-loop planning and execution, and model-based state in-ference and failure ...

The tremendous popularity of wireless systems in recent years has led to the commoditization of RF transceivers (radios) whose prices have fallen dramatically. The lower cost allows us to consider using two or more radios in the same device. Given this, we argue that wireless systems that use multiple radios in a collaborative manner dramatically improve system performance and functionality over the traditional single radio wireless systems that are popular today. In this context, we revisit some standard problems in wireless networking, including energy management, capacity enhancement, mobility management, channel failure recovery, and last-hop packet scheduling. We show that a systems approach can alleviate many of the performance and robustness issues prevalent in current wireless LAN systems. We explore the implications of the multi-radio approach on software and hardware design, as well as on algorithmic and protocol research issues. We identify three key design guidelines for constructing multi-radio systems and present results from two systems that we have built. Our experience supports our position that a multi-radio platform offers significant benefits for wireless systems. 1

In this article, we examine consumer reactions to two service recovery strategies: fixing the service failure for a fee and fixing the service failure for no fee and adding compensation. We expect that the more desirable recovery strategy will result in higher levels of satisfaction and higher postpurchase intentions. However, the effect of these recovery strategies on consumers' reactions is likely to be moderated by the setting (online versus offline). The empirical results support our predictions that the effect of the service failure recovery on satisfaction and postpurchase intentions would be stronger in offline media than in online media. Managerial implications thus are presented. D

Modern computer systems are expected to be up continuously: even planned downtime to accomplish system reconfiguration is becoming unacceptable, so more and more changes are having to be made to "live" systems that are running production workloads. One of those changes is data migration: moving data from one storage device to another for load balancing, system expansion, failure recovery, or a myriad of other reasons. Traditional methods for achieving this either require application down-time, or severely impact the performance of foreground applications -neither a good outcome when performance predictability is almost as important as raw speed. Our solution to this problem, Aqueduct, uses a control-theoretical approach to statistically guarantee a bound on the amount of impact on foreground work during a data migration, while still accomplishing the data migration in as short a time as possible. The result is better quality of service for the end users, less stress for the system administrators, and systems that can be adapted more readily to meet changing demands.

A mobile computing system consists of mobile and stationary nodes, connected to each other by a communication network. The presence of mobile nodes in the system places constraints on the permissible energy consumption and available communication bandwidth. To minimize the lost computation during recovery from node failures, periodic collection of a consistent snapshot of the system (checkpoint) is required. Locating mobile nodes contributes to the checkpointing and recovery costs. Synchronous snapshot collection algorithms, designed for static networks, either force every node in the system to take a new local snapshot, or block the underlying computation during snapshot collection. Hence, they are not suitable for mobile computing systems. If nodes take their local checkpoints independently in an uncoordinated manner, each node may have to store multiple local checkpoints in stable storage. This is not suitable for mobile nodes as they have small memory. This paper presents a synchronous snapshot collection algorithm for mobile systems that neither forces every node to take a local snapshot, nor blocks the underlying computation during snapshot collection. If a node initiates snapshot collection, local snapshots of only those nodes that have directly or transitively a ected the initiator since their last snapshots need to be taken. We prove that the global snapshot collection terminates within a nite time of its invocation and the collected global snapshot is consistent. We also propose a minimal rollback/recovery algorithm in which the computation at a node is rolled back only if it depends on operations that have been undone due to the failure of node(s). Both the algorithms have low communication and storage overheads and meet the low energy consumption and low bandwidth constraints of mobile computing systems.

This paper presents a failure detection service (FDS) and a flexible failure handling framework (Grid-WFS) as a fault tolerance mechanism on the Grid. The FDS enables the detection of both task crashes and user-defined exceptions. A major challenge in providing such a generic failure detection service on the Grid is to detect those failures without requiring any modification to both the Grid protocol and the local policy of each Grid node. This paper describes how to overcome the challenge by using a notification mechanism which is based on the interpretation of notification messages being delivered from the underlying Grid resources. The Grid-WFS built on top of FDS allows users to achieve failure recovery in a variety of ways depending on the requirements and constraints of their applications. Central to the framework is flexibility in handling failures. This paper describes how to achieve the flexibility by the use of workflow structure as a high-level recovery policy specification, which enables support for multiple failure recovery techniques, the separation of failure handling strategies from the application code, and user-defined exception handlings. Finally, this paper presents an experimental evaluation of the Grid-WFS using a simulation, demonstrating the value of supporting multiple failure recovery techniques in Grid applications to achieve high performance in the presence of failures.

Brick and object-based storage architectures have emerged as a means of improving the scalability of storage clusters. However, existing systems continue to treat storage nodes as passive devices, despite their ability to exhibit significant intelligence and autonomy. We present the design and implementation of RADOS, a reliable object storage service that can scales to many thousands of devices by leveraging the intelligence present in individual storage nodes. RADOS preserves consistent data access and strong safety semantics while allowing nodes to act semi-autonomously to selfmanage replication, failure detection, and failure recovery through the use of a small cluster map. Our implementation offers excellent performance, reliability, and scalability while providing clients with the illusion of a single logical object store.

The IEEE 802.11i wireless networking protocol provides mutual authentication between a network access point and user devices prior to user connectivity. The protocol consists of several parts, including an 802.1X authentication phase using TLS over EAP, the 4-Way Handshake to establish a fresh session key, and an optional Group Key Handshake for group communications. Motivated by previous vulnerabilities in related wireless protocols and changes in 802.11i to provide better security, we carry out a formal proof of correctness using a Protocol Composition Logic previously used for other protocols. The proof is modular, comprising a separate proof for each protocol section and providing insight into the networking environment in which each section can be reliably used. Further, the proof holds for a variety of failure recovery strategies and other implementation and configuration options. Since SSL/TLS is widely used apart from 802.11i, the security proof for SSL/TLS has independent interest.

The rapid advances in dense wavelength-division multiplexing technology with hundreds of wavelengths per fiber and worldwide fiber dcployment have brought about a tremendous increasc in the size (i.e., number of ports) of photonic cross-connects, as well as in thc cost and difficulty associated with conrrolling such largc cross-connects. Waveband Switching (WBS) has attractcd attention for its practical importance in reducing the port count, associated control complexity, and cost of photonic cross-cunnects. In this article we show that WBS is differcnt from traditional wavelength routing, and thus techniques developed for wavelengthrouted networks (including. for example, those for traffic grooming) cannot he directly applied to effectively address WBS-related problems. We dcscrihe two multigranular OXC architectures f o r WBS. By using the multilayer MG-OXC in conjunction with intelligent WBS algorithms for hoth static and dynamic traffic, we show that one can achieve considerable savings in thc port count. Wc also present various WBS schemes and lightpath grouping stratcgies, and discuss issues related to waveband conversion and failure recovery in WBS nctworks.

Precise failure analysis requires accurate fault diagnosis. A previously proposed method for diagnosing bridging faults using single stuck-at dictionaries was applied only to small circuits, produced large and imprecise diagnoses, and did not take i n to account the Byzantine Generals Problem for bridging faults. We analyze the original technique and improve i t b y introducing the concepts of match restriction, match requirement, and failure r ecovery. Our new technique, which requires no information other than that used by standard stuck-at methods, produces diagnoses that are an order of magnitude smaller than those produced by the original technique and produces many fewer misleading diagnoses than that of traditional stuck-at diagnosis.

We are interested in the validation of a cognitive theory of human communication, grounded in a speech acts perspective. The theory we refer to is outlined, and a number of predictions are drawn from it. We report a series of protocols administered to 13 brain-injured subjects and to a comparable control group. The tasks included direct and indirect speech acts, irony, deceits, failures of communication, and theory of mind inferences. All the predicted trends of difficulty are consistently verified; in particular, difficulty increases from direct/indirect speech acts to irony, from irony to deceits, and from deceits to failure recovery. This trend symmetrically shows both in the successful situation and in the failure situation. Further, failure situations prove more difficult to handle than the relevant successful situation. In sharp contrast with previous literature, there is no difference between the subjects' comprehension of direct and indirect speech acts. The results are discussed in the light of our theoretical approach.

Several distributed real-time applications (e.g., medical imaging, air traffic control, and video conferencing) demand hard guarantees on the message delivery latency and the recovery delay from component failures. As these demands cannot be met in traditional datagram services, special schemes have been proposed to provide timely recovery for real-time communications in multihop networks. These schemes reserve additional network resources (spare resources) a priori along a backup channel that is disjoint with the primary. Upon a failure in the primary channel, its backup is activated, making the real-time connection dependable. In this paper, we propose a new method of providing backups called segmented backups, in which backup paths are provided for partial segments of the primary path rather than for its entire length, as is done in the existing schemes. We show that our method offers: 1) improved network resource utilization; 2) higher average call acceptance rate; 3) better quality-of-service guarantees on propagation delays and failure-recovery times; and 4) increased flexibility to control the level of fault tolerance of each connection separately. We provide an algorithm for routing the segmented backups and prove its optimality with respect to spare resource reservation. We detail necessary extensions to resource reservation protocol (RSVP) to support our scheme and argue that they increase the implementation complexity of RSVP minimally. Our simulation studies on various network topologies demonstrate that spare resource aggregation methods such as backup multiplexing are more effective when applied to our scheme than to earlier schemes.

Routing protocols for wireless sensor networks must address the challenges of reliable packet delivery at increasingly large scale and highly constrained node resources. Attempts to limit node state can result in undesirable worst-case routing performance, as measured by stretch, which is the ratio of the hop count of the selected path to that of the optimal path.

Unknown, unexplored and abandoned subterranean voids threaten mining operations, surface developments and the environment. Hazards within these spaces preclude human access to create and verify extensive maps or to characterize and analyze the environment. To that end, we have developed a mobile robot capable of autonomously exploring and mapping abandoned mines. To operate without communications in a harsh environment with little chance of rescue, this robot must have a robust electro-mechanical platform, a reliable software system, and a dependable means of failure recovery. Presented are the mechanisms, algorithms, and analysis tools that enable autonomous mine exploration and mapping along with extensive experimental results from eight successful deployments into the abandoned Mathies coal mine near Pittsburgh, PA.

Key service elements combine to create the service concept and its value proposition for customers. During service operations failures, employee interactions with customers are a critical service element in restoring customer satisfaction. However, research in consumer psychology shows that customers seek reasons for service failures and their attributions of blame moderate the effects of the failure on the level of customer satisfaction. This paper extends research on services operations failures by hypothesizing that attributions of blame also affect what matters to the customer during service failures. Specifically, we hypothesize that the relative weights that customers assign to the key elements of the service in reaching an overall assessment of customer satisfaction are affected by customer attributions of blame for service failures. We use the U.S. airline industry as a quasi-experimental research setting to investigate the components of customer satisfaction for three samples of customers who experience: 1) routine service, 2) flight delays of external (i.e., weather) origin, and 3) flight delays of internal origin. Although the level of customer satisfaction is lower for all service failures, we find that the key components of satisfaction differ between delayed and routine flights only when customers blame the service provider for the failure. Specifically, when delays are of external original, satisfaction is lower than for routine flights, but there is virtually no difference in the weight that customers assign to the components of customer satisfaction (including employee interactions). In contrast, when delays are of internal origin, satisfaction is lower than for either routine flights or flights delayed by external factors and employee interactions have a significantly diminished role in customer satisfaction evaluations. Contrary to the popular view that employee interactions take on a greater role in determining customer satisfaction during service failures, we find that the opposite is true if the customer attributes blame to the service provider. The results highlight the important role of customer attributions during service failures and present more nuanced evidence on the role of employee-customer interactions in mitigating the effects of service failures on customer satisfaction. Data Availability: Data for replicating the results of this study are available online at: [insert web site address]. Included in an online appendix and as electronic data files are: the LISREL program code for the basic model and data files containing the variable means, standard deviations and covariance matrices for each of the three treatment groups. In addition to replicating the results of the study, the reader may explore any model that is nested within our model by making changes to the original program code to reflect constraints of a nested model. Confidentiality and nondisclosure agreements with the data provider preclude us from redistributing the raw survey data or reporting results that may be used to identify the customer satisfaction performance of any air carrier.

One of the desirable features of any network is its ability to keep services running despite a link or node failure. This ability is usually referred to as network resilience and has become a key demand from service providers. Resilient networks recover from a failure by repairing themselves automatically by diverting traffic from the failed part of the network to another portion of the network. This traffic diversion process should be fast enough to ensure that the interruption of service due to a link or node failure is either unnoticeable or as small as possible. The new path taken by a diverted traffic can be computed at the time a failure occurs through a procedure called rerouting. Alternatively the path can be computed before a failure occurs through a procedure called fast reroute. Much attention is currently being paid to fast reroute because service providers who are used to the 50-ms failure recovery time associated with SONET networks are demanding the same feature from IP and MPLS networks. While this requirement can easily be met in SONET because it operates at the physical layer, it is not easily met in IP and MPLS networks that operate above the physical layer. However, over the last few years, several schemes have been proposed for accomplishing 50-ms fast reroutes for IP and MPLS networks. The purpose of this paper is to provide a survey of the IP fast reroute and MPLS fast reroute schemes that have been proposed.

Over the past decade the number of processors in the high performance facilities went up to hundreds of thousands. As a direct consequence, while the computational power follow the trend, the mean time between failures (MTBF) suffered, and it's now being counted in hours. In order to circumvent this limitation, a number of fault tolerant algorithms as well as execution environments have been developed using the message passing paradigm. Among them, message logging has been proved to achieve a better overall performance when the MTBF is low, mainly due to it's faster failure recovery. However, message logging suffers from a high overhead when no failure occurs. Therefore, in this paper we discuss a refinement of the message logging model intended to improve failure free message logging performance. The proposed approach simultaneously removes useless memory copies and reduces the number of logged events. We present the implementation of a pessimistic message logging protocol in Open MPI and compare it with the previous reference implementation MPICH-V2. Results outline a several order of magnitude improvement on performance and a zero overhead for most messages.

This paper presents an approach to concurrency control based on the decomposition of both the database and the individual transactions. This approach is a generalization of serializability theory in that the set of permissible transaction schedules contains all the serializable schedules. In addition to providing a higher degree of concurrency than that provided by serializability theory, this approach retains three important properties associated with serializability: the consistency of the database is preserved, the individual transactions are executed correctly, and the concurrency control approach is modular, a concept formalized in this paper. The associated failure recovery procedure is also presented as is the concept of failure safety.

Advance reservation of lightpaths in Grid environments is necessary to guarantee QoS and reliability. In this paper, we have evaluated and compared several algorithms for dynamic scheduling of lightpaths using a flexible advance reservation model. The main aim is to find the best scheduling policy for a Grid network resource manager that improves network utilization and minimizes blocking. The scheduling of lightpaths involve both routing and wavelength assignment. Our simulation results show that minimum cost adaptive routing where link costs are determined by the current and future usage of the link provides the minimum blocking. For wavelength assignment, we have used a scheme that reduces fragmentation by minimizing unused gaps. We have also analyzed approaches for failure recovery and resource optimization.

Network applications of the future will require advanced mechanisms for automatic failure recovery in order to provide an acceptable quality of service. Because of this requirement, there is a need for tools that can inject simulated faults to verify a system's fault detection and recovery methods. We present a set of objectives that such a fault injection tool must meet, and describe a prototype we have developed that meets these criteria. This tool, FIG, is capable of selectively introducing and logging errors at the application/library boundary on a running system.

This paper addresses the problem of markerless tracking of a human in full 3D with a high-dimensional (29D) body model. Most work in this area has been focused on achieving accurate tracking in order to replace marker-based motion capture, but do so at the cost of relying on relatively clean observing conditions. This paper takes a different perspective, proposing a body-tracking model that is explicitly designed to handle real-world conditions such as occlusions by scene objects, failure recovery, long-term tracking, auto-initialisation, generalisation to different people and integration with action recognition. To achieve these goals, an action's motions are modelled with a variant of the hierarchical hidden Markov model. The model is quantitatively evaluated with several tests, including comparison to the annealed particle filter, tracking different people and tracking with a reduced resolution and frame rate.

A model of remote procedure call (RPC) which reflects certain generic properties of the application layer that can be exploited by the RPC layer during failure recovery is presented. A technique of adopting orphans caused by failures, which is based on the model, is described. The technique minimizes the rollback which may be required in orphan-killing techniques. Algorithmic details of the adoption technique are described, and a quantitative analysis is presented. The model is implemented as a prototype on a local area network. The simplicity and generality of the failure recovery renders the RPC model useful in distributed systems, particularly those that are large and heterogeneous and hence have complex failure modes. >

Many processes must complete in the presence of failures. Different systems respond to task failure in different ways. The system may resume a failed task from the failure point (or a saved checkpoint shortly before the failure point), it may give up on the task and select a replacement task from the ready queue, or it may restart the task. The behavior of systems under the first two scenarios is well documented, but the third (RESTART) has resisted detailed analysis. In this paper we derive tight asymptotic relations between the distribution of task times without failures to the total time when including failures, for any failure distribution. In particular, we show that if the task time distribution has an unbounded support then the total time distribution H is always heavy-tailed. Asymptotic expressions are given for the tail of H in various scenarios. The key ingredients of the analysis are the Cramér-Lundberg asymptotics for geometric sums and integral asymptotics, that in some cases are obtained via Tauberian theorems and in some cases by bare-hand calculations.

The combination of photogrammetric aerial and terrestrial recording methods can provide new opportunities for photogrammetric applications. A UAV (Unmanned Aerial Vehicle), in our case a helicopter system, can cover both the aerial and quasi-terrestrial image acquisition methods. A UAV can be equipped with an on-board high resolution camera and a priori knowledge of the operating area where to perform photogrammetric tasks. In this general scenario our paper proposes vision-based techniques for localizing a UAV. Only natural landmarks provided by a feature tracking algorithm will be considered, without the help of visual beacons or landmarks with known positions. The novel idea is to perform global localization, position tracking and localization failure recovery (kidnapping) based only on visual matching between current view and available georeferenced satellite images. The matching is based on SIFT features and the system estimates the position of the UAV and its altitude on the base of the reference image. The vision system replaces the GPS signal combining position information from visual odometry and georeferenced imagery. Georeferenced satellite or aerial images must be available on-board beforehand or downloaded during the flight. The growing availability

A sensor network can be described as a collection of sensor nodes which coordinate with each other to perform some specific function. These sensor nodes are mainly in large numbers and are densely deployed either inside the phenomenon or very close to it. They can be used for various application areas (e.g. health, military, home). Failures are inevitable in wireless sensor networks due to inhospitable environment and unattended deployment. Therefore, it is necessary that network failures are detected in advance and appropriate measures are taken to sustain network operation. We previously proposed a cellular approach for fault detection and recovery. In this paper we extend the cellular approach and propose a new fault management mechanism to deal with fault detection and recovery. We propose a hierarchical structure to properly distribute fault management tasks among sensor nodes by introducing more 'self-managing' functions. The proposed failure detection and recovery algorithm has been compared with some existing related work and proven to be more energy efficient.

Web services emergence has triggered extensive research efforts. Currently, there is a trend towards deploying business processes as an orchestration of web services compositions. Given that web services are inherently looselycoupled and are primarily built independently, they are most likely to have characteristics (e.g., transaction support, failure recovery, access policies) that might not be compliant with each other. It follows that guarantying the reliability and availability of the obtained web services compositions is a challenging issue. Aligned with this tendency, we focus on the availability and reliability of web services compositions. Specifically, in this paper, we propose THROWS, an architecture for highly available distributed execution of web services compositions. In THROWS architecture, the execution control is hierarchically delegated among dynamically discovered engines. The progress of the compositions execution by several distributed engines is continuously captured. Moreover, the web services compositions executed through the architecture we propose are previously specified as an hierarchy of arbitrary-nested transactions. These transactions execution is provided with retrial and compensation mechanisms which allow the highly available web services compositions execution.

In order to achieve resilient multipath routing we introduce the concept of Independent Directed Acyclic Graphs (IDAGs) in this study. Link-independent (Node-independent) DAGs satisfy the property that any path from a source to the root on one DAG is link-disjoint (node-disjoint) with any path from the source to the root on the other DAG. Given a network, we develop polynomial time algorithms to compute link-independent and node-independent DAGs. The algorithm developed in this paper: (1) provides multipath routing; (2) utilizes all possible edges; (3) guarantees recovery from single link failure; and (4) achieves all these with at most one bit per packet as overhead when routing is based on destination address and incoming edge. We show the effectiveness of the proposed IDAGs approach by comparing key performance indices to that of the independent trees and multiple pairs of independent trees techniques through extensive simulations.

Most research to date in survivable optical network design and operation, focused on the failure of a single component such as a link or a node. A double-link failure model in which any two links in the network may fail in an arbitrary order was proposed recently in literature . Three loop-back methods of recovering from double-link failures were also presented. The basic idea behind these methods is to pre-compute two backup paths for each link on the primary paths and reserve resources on these paths. Compared to protection methods for single-link failure model, the protection methods for double-link failure model require much more spare capacity. Reserving dedicated resources on every backup path at the time of establishing primary path itself would consume excessive resources. Moreover, it may not be possible to allocate dedicated resources on each of two backup paths around each link, due to the wavelength continuous constraint. In M. Sridharan et al., we captured the various operational phases in survivable WDM networks as a single integer programming based (ILP) optimization problem. In this work, we extend our optimization framework to include double-link failures. We use the double-link failure recovery methods available in literature, employ backup multiplexing schemes to optimize capacity utilization, and provide 100% protection guarantee for double-link failure recovery. We develop rules to identify scenarios when capacity sharing among interacting demand sets is possible. Our results indicate that for the double-link failure recovery methods, the shared-link protection scheme provides 10-15% savings in capacity utilization over the dedicated link protection scheme which reserves dedicated capacity on two backup paths for each link. We provide a way of adapting the heuristic based doublelink failure recovery methods into a mathematical framework, and use techniques to improve wavelength utilization for optimal capacity usage.

We present the availability model of a high availability SIP Application Server configuration on WebSphere. Hardware, operating system and application server failures are considered. Different types of fault detectors, detection delays, failover delays, restarts, reboots and repairs are considered. Imperfect coverages for detection, failover and recovery are incorporated. Computations are based on a set of interacting sub-models of all system components capturing their failure and recovery behavior. The parameter values used in the calculations are based on several sources, including field data, high availability testing, and agreedupon assumptions. In cases where a parameter value is uncertain, due to assumptions or limited test data, a sensitivity analysis of that parameter has been provided. Our analysis indicates the failure types and recovery parameters that are most critical in their impact on overall system availability. These results will help guide system improvement efforts throughout future releases of these products.

Current trends suggest future software systems will comprise collections of components that combine and recombine dynamically in reaction to changing conditions. Service-discovery protocols, which enable software components to locate available software services and to adapt to changing system topology, provide one foundation for such dynamic behavior. Emerging discovery protocols specify alternative architectures and behaviors, which motivate a rigorous investigation of the properties underlying their designs. Here, we assess the ability of selected designs for service-discovery protocols to maintain consistency in a distributed system during catastrophic communication failure. We use an architecture description language, called Rapide, to model two different architectures (two-party and three-party) and two different consistencymaintenance mechanisms (polling and notification). We use our models to investigate performance differences among combinations of architecture and consistency-maintenance mechanism as interface-failure rate increases. We measure system performance along three dimensions: (1) update responsiveness (How much latency is required to propagate changes?), (2) update effectiveness (What is the probability that a node receives a change?), and (3) update efficiency (How many messages must be sent to propagate a change throughout the topology?). We use Rapide to understand how failure-recovery strategies contribute to differences in performance. We also recommend improvements to architecture description languages.

Legacy application design models, which are still widely used for developing context-aware applications, incur important limitations. Firstly, embedding contextual dependencies in the form of if-then rules specifying how applications should react to context changes is impractical to accommodate the large variety of possibly even unanticipated context types and their values. Additionally, application development is complicated and challenging, as programmers have to manually determine and encode the associations of all possible combinations of context parameters with application behaviour. In this paper we propose a framework for building context aware applications on-demand, as dynamically composed sequences of calls to services. We present the design and implementation of our system, which employs goal-oriented inferencing for assembling composite services, dynamically monitors their execution, and adapts applications to deal with contex-tual changes. We describe the failure recovery mechanisms we have implemented, allowing the deployment of the system in a non-perfect environment, and avoiding the delays inherent in re-discovering a suitable service instance. By means of experimental evaluation in a realistic infotainment application, we demonstrate the potential of the proposed solution an effective, efficient, and scalable approach.

Network operators are migrating towards IP over WDM architectures. In such multi-layer networks, it is necessary to efficiently use the resources available from both layers in order to provide coordinated recovery strategies. Thanks to the development of the control plane (GMPLS and ASON), it is feasible to set up and tear down lightpaths automatically, so the WDM layer itself can support failure recovery. This paper describes a multi-layer recovery strategy in a FAN/WDM (Flow-Aware Networking/Wavelength Division Multiplexing) architecture. We propose using the EHOT (Enhanced Hold-Off Timer) algorithm to control network operation after link or node failure. Although FAN operates only on the IP level, the presented analysis shows that it is possible to ensure sufficiently low (less than 50 ms) recovery times in FAN working over an intelligent optical layer. Additionally, the paper shows the motivation for FAN networks and presents the results of carefully selected simulation experiments which allow for evaluating the duration of outages in data transmission under various conditions.

Optimistic failure recovery mechanisms are proposed as a way to provide transparent fault tolerance to distributed applications and systems. The authors identify problems that may arise when these mechanisms are applied to vast networks including many processors and spanning large geographical areas and many administrative domains. They present a technique-recovery unit gateways-that can be used to address many of these issues with existing failure recovery algorithms. This method can be applied with minimal disruption to existing transparent recovery systems, as well as to build large optimistic recovery systems while minimizing the dependency tracking overhead

With the growing scale of high performance computing platforms, fault tolerance has become a major issue. Among the various approaches for providing fault tolerance to MPI applications, message logging has been proved to tolerate higher failure rate. However, this advantage comes at the expense of a higher overhead on communications, due to latency intrusive logging of events to a stable storage. Previous work proposed and evaluated several protocols relaxing the synchronicity of event logging to moderate this overhead. Recently, the model of message logging has been refined to better match the reality of high performance network cards, where message receptions are decomposed in multiple interdependent events. According to this new model, deterministic and non-deterministic events are clearly discriminated, reducing the overhead induced by message logging. In this paper we compare, experimentally, a pessimistic and an optimistic message logging protocol, using this new model and implemented in the Open MPI library. Although pessimistic and optimistic message logging are, respectively, the most and less synchronous message logging paradigms, experiments show that most of the time their performance is comparable.

This paper presents Sonora, a platform for mobile-cloud computing. Sonora is designed to support the development and execution of continuous mobile-cloud services. To this end, Sonora provides developers with stream-based programming interfaces that coherently integrate a broad range of existing techniques from mobile, database, and distributed systems. These range from support for disconnected operation to relational and event-driven models. Sonora's execution engine is a fault-tolerant distributed runtime that supports user-facing continuous sensing and processing services in the cloud. Key features of this engine are its dynamic load balancing mechanisms, and a novel failure recovery protocol that performs checkpoint-based partial rollback recovery with selective re-execution. To illustrate the relevance and power of the stream abstraction in describing complex mobile-cloud services we evaluate Sonora's design in the context of two services. We also validate Sonora's ...

Many current approaches to software-implemented fault tolerance (SIFT) rely on process replication, which is often prohibitively expensive for practical use due to its high performance overhead and cost. The Adaptive Reconfigurable Mobile Objects of Reliability (Armor) middleware architecture offers a scalable low-overhead way to provide high-dependability services to applications. It uses coordinated multithreaded processes to manage redundant resources across interconnected nodes, detect errors in user applications and infrastructural components, and provide failure recovery. The authors describe their experiences and lessons learned in deploying Armor in several diverse fields.

Abstract— Border gateway protocol (BGP) is the standard routing protocol between various autonomous systems (AS) in the Internet. In the event of a failure, BGP may repeatedly withdraw some routes and advertise new ones until a stable state is reached. It is known that the corresponding ...

Multi-protocol label switching (MPLS) is an evolving network technology that is used to provide traffic engineering (TE) and high speed networking. Internet service providers, which support MPLS technology, are increasingly demanded to provide high quality of service (QoS) guarantees. One of the aspects of QoS is fault tolerance. It is defined as the property of a system to continue operating in the event of failure of some of its parts. Fault tolerance techniques are very useful to maintain the survivability of the network by recovering from failure within acceptable delay and minimum packet loss while efficiently utilizing network resources. In this paper, we propose a novel approach for fault tolerance in MPLS networks. Our approach uses a modified (k, n) threshold sharing scheme with multi-path routing. An IP packet entering MPLS network is partitioned into n MPLS packets, which are assigned to node/link disjoint LSPs across the MPLS network. Receiving MPLS packets from k out of n LSPs are sufficient to reconstruct the original IP packet. The approach introduces no packet loss and no recovery delay while requiring reasonable redundant bandwidth. In addition, it can easily handle single and multiple path failures.

An important research area in the workflow management domain is the adaptation of workflows to unexpected events or failures at runtime. In this paper we present a concept for dynamic and automated workflow re-planning that allows to recover from such failures. To handle the complex dynamic situation of a partially executed workflow, we propose a multi-step procedure that includes the termination of failed activities, the sound suspension of the workflow, the generation of a new process definition, and the adequate process resumption. An important aspect of the presented re-planning procedure is that all steps including the generation of a new process definition are fully automated.

In this paper, we describe a method of execution retry for bypassing software faults in messagepassing applications. Based on the techniques of cting and message logging, we demonstrate the use of message replaying and message reordering as two mechanisms for achieving localized and fast recovery. Our approach gradually increases the rollback distance and the number of affected processes when a previous retry fails, and is therore named progrssve revy. An implementation as reusable modules to provide low-cost application-level software fault tolerance is described. Examples from our experience with telecommunications software systems are given to illustrate the benefits of the scheme.

In this paper, we propose a novel Web service composition framework which dynamically accommodates various failure recovery requirements. In the proposed framework called Adaptive Failure-handling Framework (AdaFF), failure-handling submodules are prepared during the design of a composite service and some of them are systematically selected and combined with the composite Web service at runtime in accordance with the requirement of individual users. In contrast, existing frameworks cannot adapt the failure-handling behaviors to user's requirements. AdaFF rapidly delivers a composite service supporting adaptive failure handling without manual development, and contributes to a flexible composite Web service design in that service architects never care about failure handling or variable requirements of users. For proof of concept, we implement a prototype system of the AdaFF, which automatically generates a composite service instance with Web Services Business Process Execution Language (WS-BPEL) according to the users' requirement specified in XML format, and executes the generated instance on the ActiveBPEL engine.

In choosing a network service technology, a subscriber considers many features such as latency, jitter, packet loss, security, and availability. The most important feature, and usually the one that determines the final selection, is the service availability. In this article, a full spectrum of applications are studied, ranging from the minimal constraints of home networks to the rigorous demands of Industrial Ethernet Networks. This is followed by a thorough examination of Ethernet layer resilience technologies. This paper provides the resilience characteristics that are key for each class of application Ó