Virtual Disk Snapshots

Estimation of ship self-propulsion is important for the selection of the propulsion system and the main engine so that the ship can move forward with the required speed. Resistance characteristics of the vessel or the open-water performance of a propeller only are not usually enough to assess the working conditions of the ship. Both in numerical simulations and in experiments; there is a need to treat the propulsion system and the hull as a whole for a better estimation of the self-propulsion parameters. In this study, the self-propulsion points of one submarine (DARPA Suboff) and two surface piercing vessels (KCS and DTC) were obtained with methods based on computational fluid dynamics (CFD) approach. The self-propulsion points were also calculated by a classical engineering approach that makes use of the empirical relations that may be found in the literature. The results were evaluated with respect to the experiments and numerical results generated by other researchers in this field. It was found that the self-propulsion points of traditional ship forms can be very closely approximated with a classical engineering approach, given the basic geometric and the hydrostatic properties of the hull and the propeller.

Several works have shown that the time to boot one virtual machine (VM) can last up to a few minutes in high consolidated cloud scenarios. This time is critical as VM boot duration defines how an application can react w.r.t. demands’ fluctuations (horizontal elasticity). To limit as much as possible the time to boot a VM, we design the YOLO mechanism (You Only Load Once). YOLO optimizes the number of I/O operations generated during a VM boot process by relying on the boot image abstraction, a subset of the VM image (VMI) that contains data blocks necessary to complete the boot operation. Whenever a VM is booted, YOLO intercepts all read accesses and serves them directly from the boot image, which has been locally stored on fast access storage devices (e.g., memory, SSD, etc.). Creating boot images for 900+ VMIs from Google Cloud shows that only 40 GB is needed to store all the mandatory data. Experiments show that YOLO can speed up VM boot duration 2-13 times under different resourc...

Cloud Computing offers the possibility of computing resources, allowing remote access to software, storage and data processing through the Internet. Infrastructures as a Service (IaaS), it is a flexible space which can be used as an experimental environment, in which experiments can be carried out similar to a real environment, such as in a cluster can be carried out. Before making installations and changes in a production cluster or select resource in the cloud, it is important to analyze the impact of this change. For this reason we propose using the cloud to carry out the study of previous viability. In this paper, we observe the viability of using the cloud to analyze the behavior of the Checkpoint as one of the Fault Tolerance strategies, establishing the differences that exist in the information generated in a real environment (cluster) and a virtual environment (cloud). The results obtained show that due to the variability of the cloud, the impact on the benefits cannot be analyzed. However, the cloud is suitable for extracting the spatial and temporal behavior pattern of the checkpoint, which helps to characterize it and this will help us to know the right configuration and the development of methodologies and tools that simulate and predict the execution of the checkpoint in a real environment.

The scientific community is exploring the suitability of cloud infrastructure to handle High Performance Computing (HPC) applications. The goal of Magellan, a project funded through DOE ASCR, is to investigate the potential role of cloud computing to address the computing needs of the Department of Energy's Office of Science, especially for mid-range computing and data-intensive applications which are not served through existing DOE centers today. Prior work has shown that applications with significant communication or I/O tend to perform poorly in virtualized cloud environments. However, there is a limited understanding of the I/O characteristics of cloud environments. This paper will present our results in benchmarking the I/O performance over different cloud and HPC platforms to identify the major bottlenecks in existing infrastructure. We compare the I/O performance using IOR benchmarks on two cloud platforms-Amazon and the Magellan cloud testbed. We analyze the performance of different storage options available on different instance types in multiple availability zones. Finally, we do some custom benchmarking in order to analyze the variability in the I/O patterns over time and region. Our results highlight the performance of the different storage options enabling applications to make effective storage option choices.

Infrastructure as a Service (IaaS) cloud computing has revolutionized the way we think of acquiring resources by introducing a simple change: allowing users to lease computational resources from the cloud provider's datacenter for a short time by deploying virtual machines (VMs) on these resources. This new model raises new challenges in the design and development of IaaS middleware. One of those challenges is the need to deploy a large number (hundreds or even thousands) of VM instances simultaneously. Once the VM instances are deployed, another challenge is to simultaneously take a snapshot of many images and transfer them to persistent storage to support management tasks, such as suspend-resume and migration. With datacenters growing rapidly and configurations becoming heterogeneous, it is important to enable efficient concurrent deployment and snapshotting that are at the same time hypervisor independent and ensure a maximum compatibility with different configurations. This paper addresses these challenges by proposing a virtual file system specifically optimized for virtual machine image storage. It is based on a lazy transfer scheme coupled with object versioning that handles snapshotting transparently in a hypervisor-independent fashion, ensuring high portability for different configurations. Large-scale experiments on hundreds of nodes demonstrate excellent performance results: speedup for concurrent VM deployments ranges from a factor of 2 up to 25, with a reduction in bandwidth utilization of as much as 90%.

Transparent hypervisor-level checkpoint-restart mechanisms for virtual clusters (VCs) or clusters of virtual machines (VMs) offer an attractive fault tolerance capability for cloud data centers. However, existing mechanisms have suffered from high checkpoint downtimes and overheads. This paper introduces Mekha, a novel hypervisor-level, in-memory coordinated checkpoint-restart mechanism for VCs that leverages precopy live migration. During a VC checkpoint event, Mekha creates a shadow VM for each VM and employs a novel memory-bound timed-multiplex data (MTD) transfer mechanism to replicate the state of each VM to its corresponding shadow VM. We also propose a global ending condition that enables the checkpoint coordinator to control the termination of the MTD algorithm for every VM in a VC, thereby reducing overall checkpoint latency. Furthermore, the checkpoint protocols of Mekha are designed based on barrier synchronizations and virtual time, ensuring the global consistency of checkpoints and utilizing existing data retransmission capabilities to handle message loss. We conducted several experiments to evaluate Mekha using a message passing interface (MPI) application from the NASA advanced supercomputing (NAS) parallel benchmark. The results demonstrate that Mekha significantly reduces checkpoint downtime compared to traditional checkpoint mechanisms. Consequently, Mekha effectively decreases checkpoint overheads while offering efficiency and practicality, making it a viable solution for cloud computing environments.

DMTCP (Distributed MultiThreaded CheckPointing) is a transparent user-level checkpointing package for distributed applications. Checkpointing and restart is demonstrated for a wide range of over 20 well known applications, including MATLAB, Python, TightVNC, MPICH2, OpenMPI, and runCMS. RunCMS runs as a 680 MB image in memory that includes 540 dynamic libraries, and is used for the CMS experiment of the Large Hadron Collider at CERN. DMTCP transparently checkpoints general cluster computations consisting of many nodes, processes, and threads; as well as typical desktop applications. On 128 distributed cores (32 nodes), checkpoint and restart times are typically 2 seconds, with negligible run-time overhead. Typical checkpoint times are reduced to 0.2 seconds when using forked checkpointing. Experimental results show that checkpoint time remains nearly constant as the number of nodes increases on a medium-size cluster. DMTCP automatically accounts for fork, exec, ssh, mutexes/semaphores, TCP/IP sockets, UNIX domain sockets, pipes, ptys (pseudo-terminals), terminal modes, ownership of controlling terminals, signal handlers, open file descriptors, shared open file descriptors, I/O (including the readline library), shared memory (via mmap), parent-child process relationships, pid virtualization, and other operating system artifacts. By emphasizing an unprivileged, user-space approach, compatibility is maintained across Linux kernels from 2.6.9 through the current 2.6.28. Since DMTCP is unprivileged and does not require special kernel modules or kernel patches, DMTCP can be incorporated and distributed as a checkpoint-restart module within some larger package.

The scientific community is exploring the suitability of cloud infrastructure to handle High Performance Computing (HPC) applications. The goal of Magellan, a project funded through DOE ASCR, is to investigate the potential role of cloud computing to address the computing needs of the Department of Energy's Office of Science, especially for mid-range computing and data-intensive applications which are not served through existing DOE centers today. Prior work has shown that applications with significant communication or I/O tend to perform poorly in virtualized cloud environments. However, there is a limited understanding of the I/O characteristics of cloud environments. This paper will present our results in benchmarking the I/O performance over different cloud and HPC platforms to identify the major bottlenecks in existing infrastructure. We compare the I/O performance using IOR benchmarks on two cloud platforms-Amazon and the Magellan cloud testbed. We analyze the performance of different storage options available on different instance types in multiple availability zones. Finally, we do some custom benchmarking in order to analyze the variability in the I/O patterns over time and region. Our results highlight the performance of the different storage options enabling applications to make effective storage option choices.

InfiniBand is widely used for low-latency, high-throughput cluster computing. Saving the state of the InfiniBand network as part of distributed checkpointing has been a long-standing challenge for researchers. Because of a lack of a solution, typical MPI implementations have included custom checkpoint-restart services that "tear down" the network, checkpoint each node as if the node were a standalone computer, and then reconnect the network again. We present the first example of transparent, system-initiated checkpoint-restart that directly supports In-finiBand. The new approach is independent of any particular Linux kernel, thus simplifying the current practice of using a kernel-based module, such as BLCR. This direct approach results in checkpoints that are found to be faster than with the use of a checkpoint-restart service. The generality of this approach is shown not only by checkpointing an MPI computation, but also a native UPC computation (Berkeley Unified Parallel C), which does not use MPI. Scalability is shown by checkpointing 2,048 MPI processes across 128 nodes (with 16 cores per node). In addition, a cost-effective debugging approach is also enabled, in which a checkpoint image from an InfiniBand-based production cluster is copied to a local Ethernet-based cluster, where it can be restarted and an interactive debugger can be attached to it. This work is based on a plugin that extends the DMTCP (Distributed MultiThreaded CheckPointing) checkpoint-restart package.

Large-and small-scale storage systems frequently serve a mixture of workloads, an increasing number of which require some form of performance guarantee. Providing guaranteed disk performance-the equivalent of a "virtual disk"-is challenging because disk requests are nonpreemptible and their execution times are stateful, partially non-deterministic, and can vary by orders of magnitude. Guaranteeing throughput, the standard measure of disk performance, requires worst-case I/O time assumptions orders of magnitude greater than average I/O times, with correspondingly low performance and poor control of the resource allocation. We show that disk time utilizationanalogous to CPU utilization in CPU scheduling and the only fully provisionable aspect of disk performance-yields greater control, more efficient use of disk resources, and better isolation between request streams than bandwidth or I/O rate when used as the basis for disk reservation and scheduling.

The capability of taking snapshots is approaching ubiquity as a feature of file systems and data storage arrays. Here, we present an approach to structuring and managing snapshots in a storage space that provides for rapid creation and roll-back. This approach has been realized in the form of a Plan 9 kernel device that can be interposed between any pair of storage service and application. The kernel device has, in turn, been used to support an experimental file system. In this paper, we discuss the basic snapshot model, its implementation, and its application to a file system. Finally, we consider extensions to the model for supporting the deletion of snapshots.

Résumé: With exascale computing on the horizon, the performance variability of I/O systems represents a key challenge in sustaining high performance. In many HPC applications, I/O is concurrently performed by all processes, which leads to I/O bursts. This causes resource contention and substantial variability of I/O performance, which significantly impacts the overall application performance together with the predictability of its run time. In this paper, we first illustrate the presence of I/O jitter on different platforms, and show the impact of ...

The actor model is popular for many types of server applications. Efficient snapshotting of applications is crucial in the deployment of pre-initialized applications or moving running applications to different machines, e.g for debugging purposes. A key issue is that snapshotting blocks all other operations. In modern latency-sensitive applications, stopping the application to persist its state needs to be avoided, because users may not tolerate the increased request latency. In order to minimize the impact of snapshotting on request latency, our approach persists the application's state asynchronously by capturing partial heaps, completing snapshots step by step. Additionally, our solution is transparent and supports arbitrary object graphs. We prototyped our snapshotting approach on top of the Truffle/Graal platform and evaluated it with the Savina benchmarks and the Acme Air microservice application. When performing a snapshot every thousand Acme Air requests, the number of slow requests (0.007% of all requests) with latency above 100ms increases by 5.43%. Our Savina microbenchmark results further detail how different utilization patterns impact snapshotting cost. To the best of our knowledge, this is the first system that enables asynchronous snapshotting of actor applications and minimizes the impact on latency. We thus believe it enables new deployment and debugging options for actor systems.

Distributed Virtual Disk Storage System (DVDSS) is a reliable and fully decentralized storage system which is based on the concept of Distributed Storage and Virtual Disk. It is Client / Server architecture and utilizes all free space of desktop machines of a LAN for storage purposes. DVDSS converts all storage of computers on the LAN into a large disk, create a virtual disk and provide this disk to all users of the system. Users can store their data files and folders on this disk and they can retrieve their data whenever they need. DVDSS provides a transparent and reliable access to data. It also provides data consistency and data security. The recovery of data when a node fails is also provided by DVDSS. The objective of system is to combine the all storage space of a LAN and utilize it for storage of user's data in transparent and reliable manner.

The storage needs of users have shifted from just needing to store data to requiring a rich interface which enables the efficient query of versions, snapshots and creation of clones. Providing these features in a distributed file system while maintaining scalability, strong consistency and performance remains a challenge. In this paper we introduce Silver, a file system which leverages the Corfu distributed logging system to not only store data, but to provide fast strongly consistent snapshots, clones and multi-versioning while preserving the scalability and performance of the distributed shared log. We describe and implement Silver using a FUSE prototype and show its performance characteristics.

As more and more large-scale applications need to generate and process very large volumes of data, the need for adequate storage facilities is growing. It becomes crucial to efficiently and reliably store and retrieve large sets of data that may be shared at the global scale. Based on previous systems for global data sharing (global file systems, grid data-sharing services), this paper proposes a hierarchical approach for grid storage, which combines the access efficiency of RAM storage with the scalability and persistence of the global file system approach. Our proposal has been validated through a prototype that couples the GFARM file system with the JUXMEM data-sharing service. Experiments on the Grid'5000 testbed confirm the advantages of our approach.

Blizzard is a high-performance block store that exposes cloud storage to cloud-oblivious POSIX and Win32 applications. Blizzard connects clients and servers using a network with full-bisection bandwidth, allowing clients to access any remote disk as fast as if it were local. Using a novel striping scheme, Blizzard exposes high disk parallelism to both sequential and random workloads; also, by decoupling the durability and ordering requirements expressed by flush requests, Blizzard can commit writes out-of-order, providing high performance and crash consistency to applications that issue many small, random IOs. Blizzard's virtual disk drive, which clients mount like a normal physical one, provides maximum throughputs of 1200 MB/s, and can improve the performance of unmodified, cloud-oblivious applications by 2x-10x. Compared to EBS, a commercially available, state-of-the-art virtual drive for cloud applications, Blizzard can improve SQL server IOp rates by seven-fold while still providing crash consistency. * Work completed as a Microsoft intern; now at Google. † Work completed as a Microsoft intern; now at NEC Labs. ‡ Work completed as a Microsoft intern. § Work completed as a Microsoft intern; now at Twitter.

Infrastructure-as-a-Service (IaaS) cloud computing is gaining significant interest in industry and academia as an alternative platform for running HPC applications. Given the need to provide fault tolerance, support for suspend-resume and offline migration, an efficient Checkpoint-Restart mechanism becomes paramount in this context. We propose BlobCR, a dedicated checkpoint repository that is able to take live incremental snapshots of the whole disk attached to the virtual machine (VM) instances. BlobCR aims to minimize the performance overhead of checkpointing by persisting VM disk snapshots asynchronously in the background using a low overhead technique we call selective copy-on-write. It includes support for both application-level and process-level checkpointing, as well as support to roll back file system changes. Experiments at large scale demonstrate the benefits of our proposal both in synthetic settings and for a reallife HPC application.

In this paper we propose a flexible and scalable distributed storage framework called flexStore that can adapt to variations in available or consumable power and demonstrate its performance in the context of deduplicated virtual machine disks. We propose and investigate smart control techniques in order to cope with the power constraints either introduced as a result of increasing node density in the storage arrays (consumable power constraints) or introduced when a mix of renewable (green) and conventional (brown) energy sources are used to power the datacenter. The key component in the proposed storage framework is the policy engine which is a software layer that provides interfaces to define performance requirements of the applications (and also energy related policies). The policy engine enforces those policies in the storage system by adjusting the allocation of storage resources. The experimental results demonstrate the ability of the framework to dynamically adapt to the changes in workload and power constraints and minimize performance impacts. Our evaluation of the prototype shows that the adaptive replication mechanisms can reduce the IO latencies by around 65% during energy plenty situations and the impact of adaptation actions on IO latencies during energy constrained situations is reduced by more than 40% compared to the case without the adaptive replication and optimized adaptation mechanisms.

Using the cloud computing paradigm, a host of companies promise to make huge compute resources available to users on a pay-as-you-go basis. These resources can be configured on the fly to provide the hardware and operating system of choice to the customer on a large scale. While the current target market for these resources in the commercial space is web development/hosting, this model has the lure of savings of ownership, operation, and maintenance costs, and thus sounds like an attractive solution for people who currently invest millions to hundreds of millions of dollars annually on High Performance Computing (HPC) platforms in order to support large-scale scientific simulation codes. Given the current interconnect bandwidth and topologies utilized in these commercial offerings, however, the only current viable market in HPC would be small-memoryfootprint embarrassingly parallel or loosely coupled applications, which inherently require little to no inter-processor communication. While providing the appropriate resources (bandwidth, latency, memory, etc.) for the HPC community would increase the potential to enable HPC in cloud environments, this would not address the need for scalability and reliability, crucial to HPC applications. Providing for these needs is particularly difficult in commercial cloud offerings where the number of virtual resources can far outstrip the number of physical resources, the resources are shared among many users, and the resources may be heterogeneous. Advanced resource monitoring, analysis, and configuration tools can help address these issues, since they bring the ability to dynamically provide and respond to information about the platform and application state and would enable more appropriate, efficient, and flexible use of the resources key to enabling HPC. Additionally such tools could be of benefit to non-HPC cloud providers, users, and applications by providing more efficient resource utilization in general.

A non-invasive, cloud-agnostic approach is demonstrated for extending existing cloud platforms to include checkpoint-restart capability. Most cloud platforms currently rely on each application to provide its own fault tolerance. A uniform mechanism within the cloud itself serves two purposes: (a) direct support for long-running jobs, which would otherwise require a custom fault-tolerant mechanism for each application; and (b) the administrative capability to manage an over-subscribed cloud by temporarily swapping out jobs when higher priority jobs arrive. An advantage of this uniform approach is that it also supports parallel and distributed computations, over both TCP and InfiniBand, thus allowing traditional HPC applications to take advantage of an existing cloud infrastructure. Additionally, an integrated health-monitoring mechanism detects when longrunning jobs either fail or incur exceptionally low performance, perhaps due to resource starvation, and proactively suspends the job. The cloud-agnostic feature is demonstrated by applying the implementation to two very different cloud platforms: Snooze and OpenStack. The use of a cloud-agnostic architecture also enables, for the first time, migration of applications from one cloud platform to another.

An advanced and established infrastructure for the data-center facets of Computer Science Large-scale, distributed Shared (many involved laboratories and institutions) Designed for reconfigurability, observability, reproducible research Looking forward to extend this scope through collaborations in the context of SILECS Compared with FIT: we share the same design goals, principles, but focus on different objects Core of the Internet vs Edge of the Internet Internet of servers vs Internet of clients Strong needs of joint experiments Lucas Nussbaum SILECS/Grid'5000 19 / 19

In cloud storage systems, users must be able to shut down the application when not in use and restart it from the last consistent state when required. BlobSeer is a data storage application, specially designed for distributed systems, that was built as an alternative solution for the existing popular open-source storage system-Hadoop Distributed File System (HDFS). In a cloud model, all the components need to stop and restart from a consistent state when the user requires it. One of the limitations of BlobSeer DFS is the possibility of data loss when the system restarts. As such, it is important to provide a consistent start and stop state to BlobSeer components when used in a Cloud environment to prevent any data loss. In this paper, we investigate the possibility of BlobSeer providing a consistent state distributed data storage system with the integration of checkpointing restart functionality. To demonstrate the availability of a consistent state, we set up a cluster with multipl...

Distributed Virtual Disk Storage System (DVDSS) is a reliable and fully decentralized storage system which is based on the concept of Distributed Storage and Virtual Disk. It is Client / Server architecture and utilizes all free space of desktop machines of a LAN for storage purposes. DVDSS converts all storage of computers on the LAN into a large disk, create a virtual disk and provide this disk to all users of the system. Users can store their data files and folders on this disk and they can retrieve their data whenever they need. DVDSS provides a transparent and reliable access to data. It also provides data consistency and data security. The recovery of data when a node fails is also provided by DVDSS. The objective of system is to combine the all storage space of a LAN and utilize it for storage of user's data in transparent and reliable manner.

This paper presents the image server of the VISOR cloud agnostic virtual machine images management service. An evaluation approach is also described and the results are discussed. VISOR is not intended to fit in a specific cloud framework but rather to overreach sharing and interoperability limitations among different frameworks. Such feature is achieved by relying on a flexible image metadata schema and in a storage abstraction layer, where the latter seamlessly abstracts the heterogeneity of distinct cloud storage systems into which images can be saved. VISOR is an open source highly distributed system composed of several independent web services. In its development we have focused in modularity, extensibility and performance. Conducted evaluation tests have shown encouraging results regarding performance and scalability.

Existing approaches to providing high availability (HA) for virtualized environments require a backup VM for every primary running VM. These approaches are expensive in memory because the backup VM requires the same amount of memory as the primary, even though it is normally passive. In this paper, we propose a storage-based, memory-efficient HA solution for VMs, called HydraVM, that eliminates the passive memory reservations for backups. HydraVM maintains a complete, recent image of each protected VM in shared storage using an incremental checkpointing technique. Upon failure of a primary VM, a backup can be promptly restored on any server with available memory. Our evaluation results have shown that HydraVM provides protection for VMs at a low overhead, and can restore a failed VM within 1.6 seconds without excessive use of memory resource in a virtualized environment.

In a scientific community that increasingly relies upon High Performance Computing (HPC) for large scale simulations and analysis, the reliability of hardware and applications devoted to HPC is extremely important. While hardware reliability is not likely to dramatically increase in the coming years, software must be able to provide the reliability required by demanding applications. One way to increase the reliability of HPC systems is to use checkpointing to save the state of an application. If the application fails for some reason (hardware or software errors), the application can be restarted from the most recent checkpoint. This paper presents Dynamic Virtual Clustering as a platform to enable completely transparent parallel checkpointing. 1 Dynamic Virtual Clustering As High Performance Computing (HPC) systems grow larger and more complex, reliability is an ever more important issue. At ASU, the Dynamic Virtual Clustering (DVC) system has been under development to abstract phy...

A key enabler for standardized cloud services is the encapsulation of software and data into VM images. With the rapid evolution of the cloud ecosystem, the number of VM images is growing at high speed. These images, each containing gigabytes or tens of gigabytes of data, create heavy disk and network I/O workloads in cloud data centers. Because these images contain identical or similar OS, middleware, and applications, there are plenty of data blocks with duplicate content among the VM images. However, current deduplication techniques cannot efficiently capitalize on this content similarity due to their warmup delay, resource overhead and algorithmic complexity. We propose an instant, non-intrusive, and lightweight I/O optimization layer tailored for the cloud: V irtual M achine I/O Access Redirection (VMAR). VMAR generates a block translation map at VM image creation / capture time, and uses it to redirect accesses for identical blocks to the same filesystem address before they reach the OS. This greatly enhances the cache hit ratio of VM I/O requests and leads to up to 55% performance gains in instantiating VM operating systems (48% on average), and up to 45% gain in loading application stacks (38% on average). It also reduces the I/O resource consumption by as much as 70%.

Blizzard is a high-performance block store that exposes cloud storage to cloud-oblivious POSIX and Win32 applications. Blizzard connects clients and servers using a network with full-bisection bandwidth, allowing clients to access any remote disk as fast as if it were local. Using a novel striping scheme, Blizzard exposes high disk parallelism to both sequential and random workloads; also, by decoupling the durability and ordering requirements expressed by flush requests, Blizzard can commit writes out-of-order, providing high performance and crash consistency to applications that issue many small, random IOs. Blizzard’s virtual disk drive, which clients mount like a normal physical one, provides maximum throughputs of 1200 MB/s, and can improve the performance of unmodified, cloud-oblivious applications by 2x–10x. Compared to EBS, a commercially available, state-of-the-art virtual drive for cloud applications, Blizzard can improve SQL server IOp rates by seven-fold while still prov...

With increasing interest among mainstream users to run HPC applications, Infrastructure-as-a-Service (IaaS) cloud computing platforms represent a viable alternative to the acquisition and maintenance of expensive hardware, often out of the financial capabilities of such users. Also, one of the critical needs of HPC applications is an efficient, scalable and persistent storage. Unfortunately, storage options proposed by cloud providers are not standardized and typically use a different access model. In this context, the local disks on the compute nodes can be used to save large data sets such as the data generated by Checkpoint-Restart (CR). This local storage offers high throughput and scalability but it needs to be combined with persistency techniques, such as block replication or erasure codes. One of the main challenges that such techniques face is to minimize the overhead of performance and I/O resource utilization (i.e., storage space and bandwidth), while at the same time guar...

Reed-Solomon erasure codes provide efficient simple techni ques for re- dundantly encoding information so that the failure of a few disks in a disk array doesn't compromise the availability of data. This pap er presents a tech- nique for constructing a code that can correct up to three err ors with a sim- ple, regular encoding, which admits very efficient matrix in versions. It also presents new techniques for efficiently computing the sums o f streams of data elements in a finite field, such as is needed in recovering data after a disk failure.

Reed-Solomon erasure codes provide efficient simple techni ques for re- dundantly encoding information so that the failure of a few disks in a disk array doesn't compromise the availability of data. This pap er presents a tech- nique for constructing a code that can correct up to three err ors with a sim- ple, regular encoding, which admits very efficient matrix in versions. It also presents new techniques for efficiently computing the sums o f streams of data elements in a finite field, such as is needed in recovering data after a disk failure.

In desktop grids the use of off-the-shelf shared components makes the use of dedicated resources economically nonviable and increases the complexity of design of efficient storage systems that are required to address the exponentially growing storage demands of modern applications that run on these platforms. To address this challenge, we present PeerStripe, a storage system that transparently distributes files to storage space contributed by participants that have joined a peer-to-peer (p2p) network. PeerStripe uses structured p2p routing to yield a scalable, robust, reliable, and self-organizing storage system. The novelty of PeerStripe lies in its ingenious use of striping and error coding techniques in a heterogeneous distributed environment to store very large data files. Our evaluation of PeerStripe shows that it can achieve acceptable performance for applications in desktop grids.

In a Fine-Grained Cycle Sharing (FGCS) system [1], machine owners voluntarily share their unused CPU cycles with guest jobs, as long as their performance degradation is tolerable. However, for guest users, these free computation resources come at the cost of fluctuating availability due to resource contention or resource revocation by the owners. As the size and the boundary of a grid expand, so grows the frequency of failures. The completion times of the long running compute-bound jobs fluctuate widely due to this effect. To achieve high performance in the presence of resource volatility, checkpointing and rollback [2] has been widely applied. The research questions that we investigate in this work are: • Where do we store the checkpoints and how does this affect the overall performance of the guest applications? • How does a reliable storage repository for these checkpoints improve the performance of the guest jobs? • How to harness the power of multiple cores available on grid re...

In a Fine-Grained Cycle Sharing (FGCS) system [1], machine owners voluntarily share their unused CPU cycles with guest jobs, as long as their performance degradation is tolerable. However, for guest users, these free computation resources come at the cost of fluctuating availability due to resource contention or resource revocation by the owners. As the size and the boundary of a grid expand, so grows the frequency of failures. The completion times of the long running compute-bound jobs fluctuate widely due to this effect. To achieve high ...

Abstract—As failure rate keeps on increasing in large systems, applications running atop restart more frequently than ever. Existing research on checkpoint/restart mainly focuses on optimizing checkpoint operation, without paying much attention to the restart operation. As a result, application restart latency maybe substantial, which greatly threatens system dependability and performance. To attack the restart latency problem, in this paper, we present FREM, a fast restart mechanism for general checkpoint/restart protocols. By dynamically tracking the process data accesses after each checkpoint, FREM masks restart latency by overlapping application recovery with the retrieval of its checkpoint image. We have implemented FREM as a prototype system and tested it under Linux environments. Extensive experiments with real applications demonstrate that it can effectively reduce restart latency by over 50 percent on average, as compared to the conventional restart mechanisms. Index Terms—...

Blizzard is a high-performance block store that exposes cloud storage to cloud-oblivious POSIX and Win32 applications. Blizzard connects clients and servers using a network with full-bisection bandwidth, allowing clients to access any remote disk as fast as if it were local. Using a novel striping scheme, Blizzard exposes high disk parallelism to both sequential and random workloads; also, by decoupling the durability and ordering requirements expressed by flush requests, Blizzard can commit writes out-of-order, providing high performance and crash consistency to applications that issue many small, random IOs. Blizzard's virtual disk drive, which clients mount like a normal physical one, provides maximum throughputs of 1200 MB/s, and can improve the performance of unmodified, cloud-oblivious applications by 2x-10x. Compared to EBS, a commercially available, state-of-the-art virtual drive for cloud applications, Blizzard can improve SQL server IOp rates by seven-fold while still providing crash consistency. * Work completed as a Microsoft intern; now at Google. † Work completed as a Microsoft intern; now at NEC Labs. ‡ Work completed as a Microsoft intern. § Work completed as a Microsoft intern; now at Twitter.

Blizzard is a high-performance block store that exposes cloud storage to cloud-oblivious POSIX and Win32 applications. Blizzard connects clients and servers using a network with full-bisection bandwidth, allowing clients to access any remote disk as fast as if it were local. Using a novel striping scheme, Blizzard exposes high disk parallelism to both sequential and random workloads; also, by decoupling the durability and ordering requirements expressed by flush requests, Blizzard can commit writes out-of-order, providing high performance and crash consistency to applications that issue many small, random IOs. Blizzard’s virtual disk drive, which clients mount like a normal physical one, provides maximum throughputs of 1200 MB/s, and can improve the performance of unmodified, cloud-oblivious applications by 2x–10x. Compared to EBS, a commercially available, state-of-the-art virtual drive for cloud applications, Blizzard can improve SQL server IOp rates by seven-fold while still prov...

Cloud computing in its many forms has become the key computing-infrastructure that supports business and more recently governmental computing across the globe. With its geographical spread and value proposition comes the need to provide guaranteed level of availability in the infrastructure and in its services. Multicore processing, virtualization, distributed storage systems and an overarching management framework that enable a Cloud, offer a plethora of possibilities to provide high availability using commodity systems. Herein lie the opportunities and challenges discussed in this paper.

Reed-Solomon erasure codes provide efficient simple techniques for redundantly encoding information so that the failure of a few disks in a disk array doesn’t compromise the availability of data. This paper presents a technique for constructing a code that can correct up to three errors with a simple, regular encoding, which admits very efficient matrix inversions. It also presents new techniques for efficiently computing the sums of streams of data elements in a finite field, such as is needed in recovering data after a

Comment utiliser MonALISA pour surveiller la plate-forme de gestion de données réparties BlobSeer Résumé : La surveillance des grilles est un domaine actif de la recherche, visant à la fois la surveillance des ressources et la surveillance des applications sur les grilles. La cible de la plupart des systèmes de surveillance est de produire des informations génériques. Toutefois, les applications réparties peuvent bénéficier d'un outil de surveillance et visualisation adapté à leurs besoins particuliers. Cet article étudie BlobSeer, un système de stockage réparti à large échelle. Ce travail repose sur le système MonALISA pour collecter les données génériques et spécifiques. Afin de répondre aux défis de BlobSeer, l'outil de visualisation doit être capable de passer à l'échelle, tout en étant en mésure de fournir une image détaillée et significative des données stockées.

Dealing with the large amounts of data generated by longrunning parallel applications is one of the most challenging aspects of Grid Computing. Periodic checkpoints might be taken to guarantee application progression, producing even more data. The classical approach is to employ highthroughput checkpoint servers connected to the computational nodes by high speed networks. In the case of Opportunistic Grid Computing, we do not want to be forced to rely on such dedicated hardware. Instead, we want to use the shared Grid nodes to store application data in a distributed fashion. In this work, we evaluate several strategies to store checkpoints on distributed non-dedicated repositories. We consider the tradeoff among computational overhead, storage overhead, and degree of fault-tolerance of these strategies. We compare the use of replication, parity information, and information dispersal (IDA). We used InteGrade, an objectoriented Grid middleware, to implement the storage strategies and perform evaluation experiments.

This article evaluates several strategies for storing checkpoint data in an opportunistic grid environment, including replication, parity information, and erasure coding. This evaluation compares the computational overhead, storage overhead, and degree of fault tolerance of these strategies.

Cloud computing in its many forms has become the key computing-infrastructure that supports business and more recently governmental computing across the globe. With its geographical spread and value proposition comes the need to provide guaranteed level of availability in the infrastructure and in its services. Multicore processing, virtualization, distributed storage systems and an overarching management framework that enable a Cloud, offer a plethora of possibilities to provide high availability using commodity systems. Herein lie the opportunities and challenges discussed in this paper.