BoomFS: A Declarative Approach To Building Distributed File Systems

This work describes the development of a flexible and adaptable distributed file system model where the main concepts of distributed computing are intrinsically incor-porated. The file system incorporates characteristics such as transparency, scalability, fault-tolerance, cryptography, support for low-cost hardware, easy configuration and file manipulation.

IRJET, 2020

Distributed system plays an important role in analysis and processing of huge amount of data. It is used for large number of applications such as forecasting, real time data analysis, and various other scientific application. Distributed File Systems has emerged as one of the most effective and convenient solutions to the issue of handling computations with increase in the quantity of data. Distributed file system (DFS) is mainly aimed to enable clients to use a common file system which paves the way for sharing data and resources over physically distributed machines. Many distributed computing projects have tried to implement and design distributed file systems with distinct architectures and have provided with various functionalities with them. This paper tries to build a taxonomy to understand and analyze distributed file system architecture and uses them to describe the implementation of existing distributed file systems.

In recent years, a great deal of work has been done on distributed and parallel file systems. Although the motivations behind these two types of file systems are quite different, they share many underlying characteristics and techniques. Distributed file systems need to provide clients with efficient, usable, and scalable high-bandwidth access to stored data. Parallel file systems have a somewhat more straightforward goal: to allow multiple processes working together as a single task force to access a shared file or files in parallel. In these specialized systems, speed and scalability are the most important considerations, while issues like portability and ease of use are secondary. This paper will examine eight modern distributed and parallel file systems, looking at the design principles and goals behind each, their actual implementations, the advantages and disadvantages each entails, the innovations each system introduces, and the performance and scalability each offers. The paper concludes with a brief comparison of these systems.

2006

File system designers continue to look to new architectures to improve scalability. Object-based storage diverges from server-based (e. g. NFS) and SAN-based storage systems by coupling processors and memory with disk drives, delegating low-level allocation to object storage devices (OSDs) and decoupling I/O (read/write) from metadata (file open/close) operations. Even recent object-based systems inherit decades-old architectural choices going back to early UNIX file systems, however, limiting their ability to effectively scale to hundreds of petabytes. We present Ceph, a distributed file system that provides excellent performance and reliability with unprecedented scalability. Ceph maximizes the separation between data and metadata management by replacing allocation tables with a pseudo-random data distribution function (CRUSH) designed for heterogeneous and dynamic clusters of unreliable OSDs. We leverage OSD intelligence to distribute data replication, failure detection and recovery with semi-autonomous OSDs running a specialized local object storage file system (EBOFS). Finally, Ceph is built around a dynamic distributed metadata management cluster that provides extremely efficient metadata management that seamlessly adapts to a wide range of general purpose and scientific computing file system workloads. We present performance measurements under a variety of workloads that show superior I/O performance and scalable metadata management (more than a quarter million metadata ops/sec).

Peer-to-peer storage systems organize a collection of symmetric nodes to store data without the use of centralized data structures or operations. As a result, they can scale to many thousands of nodes and can be formed directly by clients. This paper describes the design and implementation Mammoth, which implements a traditional UNIX-like hierarchical file system in a peer-to-peer fashion. Each Mammoth node stores a potentially arbitrary collection of directories and files that may be replicated on multiple nodes. The only thing that links these nodes together is that each metadata object encodes the network addresses of the nodes that store it. Data is replicated by a background process whose operation is simplified by the fact that files are stored as journals of immutable versions. An optimistic replication technique is used to allow nodes to read and write whatever version of data they can access, while also ensuring consistency when nodes are connected. In the event of temporary failure, eventual consistency is achieved by ensuring that every replica of a directory or file metadata object receives all updates to the object, irrespective of delivery order. While an update is being propagated every node that receives it cooperates to ensure that the update is delivered, even if the original sender fails. Our prototype is implemented as a user-level NFS server. Its performance is comparable to a standard NFS server and it will be publicly available soon.

Unpublished

MojaveFS is a key component of an infrastructure supporting distributed transactional computing. It is closely tied to the Mojave Compiler Collection, which provides high-level programming language constructs for reliable distributed programming. Among the most important features of the system are transparency, consistency, and fault-tolerance. We use group communication systems to ensure the consistency and serializability of all operations. The file system provides a global uniform namespace which allows for mobile computing. Transactions are supported through a journalling mechanism, and replication provides fault tolerance. The design provides a UNIX-style file system.

1996

Echo is an ambitious distributed file system. It was designed around a truly global name space. It uses a coherent caching algorithm. It is fault tolerant. And it is real—it was the primary file system for a large group of researchers. Its novel aspects include an extensible " junction " mechanism for global naming; extensive write-behind with ordering semantics that allow applications to maintain invariants without resorting to synchronous writes; and fault tolerance mechanisms that are highly configurable and that tolerate network partitions. It was designed with the intention that its performance could be as good as a local file system, while supporting large numbers of clients per server. Its reliability was designed to be higher than other distributed file systems, and higher than centralized systems. It was designed to work well in arbitrarily large networks.

[1991] Proceedings. 11th International Conference on Distributed Computing Systems, 1991

Several existing distributed file systems provide reliability by server replication. A n alternative approach is to use dual-ported disks accessible to a server and a backup. We compare the two approaches by examining an example of each. Deceit is a replicated file server that emphasizes flexibility. HA-NFS is an example of the second approach that emphasizes efficiency and simplicity. The two file servers run on the same hardware and implement SUN'S NFS protocol. The comparison shows that replicated servers are more jlexible and tolerant of a wider variety of faults. On the other hand, the dual-ported disks approach is more efficient and simpler to implement. When tolerating single failure, dual-ported disks also give somewhat better availability. 616 CH2996-7/91/0000/0616$01 .OO 0 1991 IEEE

IEEE Transactions on Computers, 1990

Abstmct-Coda is a file system for a large-scale distributed computing environment composed of Unix workstations. It provides resiliency to server and network failures through the use of two distinct but complementary mechanisms. One mechanism, server replication, stores copies of a file at multiple servers. The other mechanism, disconnected operution, is a mode of execution in which a caching site temporarily assumes the role of a replication site. Disconnected operation is particularly useful for supporting portable workstations. The design of Coda optimizes for availability and performance, and strives to provide the highest degree of consistency attainable in the light of these objectives. Measurements from a prototype show that the performance cost of providing high availability in Coda is reasonable.

2009

WheelFS is a wide-area distributed storage system intended to help multi-site applications share data and gain fault tolerance. WheelFS takes the form of a distributed file system with a familiar POSIX interface. Its design allows applications to adjust the tradeoff between prompt visibility of updates from other sites and the ability for sites to operate independently despite failures and long delays. WheelFS allows these adjustments via semantic cues, which provide application control over consistency, failure handling, and file and replica placement.