A COMPARATIVE STUDY ON DISTRIBUTED FILE SYSTEMS

In recent years, a great deal of work has been done on distributed and parallel file systems. Although the motivations behind these two types of file systems are quite different, they share many underlying characteristics and techniques. Distributed file systems need to provide clients with efficient, usable, and scalable high-bandwidth access to stored data. Parallel file systems have a somewhat more straightforward goal: to allow multiple processes working together as a single task force to access a shared file or files in parallel. In these specialized systems, speed and scalability are the most important considerations, while issues like portability and ease of use are secondary. This paper will examine eight modern distributed and parallel file systems, looking at the design principles and goals behind each, their actual implementations, the advantages and disadvantages each entails, the innovations each system introduces, and the performance and scalability each offers. The paper concludes with a brief comparison of these systems.

The Hadoop Distributed File System (HDFS) is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. We describe the architecture of HDFS and report on experience using HDFS to manage 25 petabytes of enterprise data at Yahoo!.

Distributed Operating Systems, 2010

The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications. Clients talk to a service that, in this case, happens to be a file service. The actual file service is offered by a single machine. Where multiple machines are used, the administrator is responsible for diving up file resources (e.g., home directories) among individual servers and the the entire resource (e.g., your home directory) is fully served from a single server. In Coda, which supports replication, it is up to the administrator to define the set of machines that hold volume replicas. This is not the kind of approach that we want to use if we want to divide our files among thousands or tens of thousands of machines. For an example of handling this environment, we will look at two closely-related file systems: the Google File System (GFS) and the Hadoop Distributed File System (HDFS). The latter is an open source version (and minor variant) of the former.

IRJET, 2020

Distributed system plays an important role in analysis and processing of huge amount of data. It is used for large number of applications such as forecasting, real time data analysis, and various other scientific application. Distributed File Systems has emerged as one of the most effective and convenient solutions to the issue of handling computations with increase in the quantity of data. Distributed file system (DFS) is mainly aimed to enable clients to use a common file system which paves the way for sharing data and resources over physically distributed machines. Many distributed computing projects have tried to implement and design distributed file systems with distinct architectures and have provided with various functionalities with them. This paper tries to build a taxonomy to understand and analyze distributed file system architecture and uses them to describe the implementation of existing distributed file systems.

2010

Abstract: Large scale distributed systems such as cloud computing applications are becoming very common. These applications come with increasing challenges on how to transfer and where to store and compute data. The most prevalent distributed file systems to deal with these challenges is the Hadoop File System (HDFS) which is a variant of the Google File System (GFS). However HDFS has two potential problems.

2003

We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients.

ACM Computing Surveys, 1990

The purpose of a distributed file system (DFS) is to allow users of physically distributed computers to share data and storage resources by using a common file system. A typical configuration for a DFS is a collection of workstations and mainframes connected by a local area network (LAN). A DFS is implemented as part of the operating system of each of the connected computers. This paper establishes a viewpoint that emphasizes the dispersed structure and decentralization of both data and control in the design of such systems. It defines the concepts of transparency, fault tolerance, and scalability and discusses them in the context of DFSs. The paper claims that the principle of distributed operation is fundamental for a fault tolerant and scalable DFS design. It also presents alternatives for the semantics of sharing and methods for providing access to remote files. A survey of contemporary UNIX@-based systems, namely, UNIX United, Locus, Sprite, Sun's Network File System, and ITC's Andrew, illustrates the concepts and demonstrates various implementations and design alternatives. Based on the assessment of these systems, the paper makes the point that a departure from the approach of extending centralized file systems over a communication network is necessary to accomplish sound distributed file system design.

International Journal of Computer Applications

Distributed File System (DFS) is acting as an extension to file system which manages files and data on multiple storage devices and provides more performance and reliability using various modern techniques. Outside world only sees the distributed file system as a single storage device and it is nothing but an interface to a great extent. In case of failure or heavy load very few Distributed file systems provide location transparency and redundancy to improve the data availability. Significant challenges for such a distributed file system are extended to a large number of storage nodes and providing reasonably. In this paper, there is been a few performance collaborative parameters have been studied viz. replication, fault tolerance and load balancing experimentally on various DFS.

Procedia Computer Science, 2016

There is a lot of data generated by the network is growing every day. MapReduce is a promising parallel programming model for processing large data.In this paper we surveyed several distributed storage and computation systems. We have studied various parameters like Fault Tolerance,Replication,Checkpointing,Security and Optimizing small file access using MapReduce and reviewed the distributed file systems like GlusterFS,Lustre,Ceph and HDFS which are open source distributed file systems. Cloud computing is one of the distirbuted file system which plays a very important role in protecting applications'data and the related infrastructure with the help of policies, technologies, controls, and big data tools. In our study we have proposed that MapReduce is the efficient and scalable programming platform for data processing which provides computational capabilities and distributed storage on clusters of commodity hardware.