A Flexible and Adaptable Distributed File System

2020

Data collection in the world are growing and expanding. This capability, it is important that the infrastructure must be able to store a huge collection of data on their number grows every day. The conventional methods are used in data centers for capacity building, costs of software, hardware and management of this process will be very high, today. File system architecture is that, independent of the data stored in its metadata. All of Distributed File System (DFS) are used for processing, storing and analyzing very large amount of unstructured data. The most common file system can be noted Google File System (GFS) and Hadoop Distributed File System (HDFS). In this paper, we present a review on design of distributed file system. File systems compared by scalability, availability, compatibility, extensibility, autonomy and etc.

1985

This paper presents the design and rationale of a distributed file system for a network of more than 5000 personal computer workstations. While scale has been the dominant design influence, careful attention has also been paid to the goals of location transparency, user mobility and compatibility with existing operating system interfaces. Security is an important design consideration, and the mechanisms for it do not assume that the workstations or the network are secure. Caching of entire files at workstations is a key element in this design. A prototype of this system has been built and is in use by a user community of about 400 individuals. A refined implementation that will scale more gracefully and provide better performance is close to completion.

In recent years, a great deal of work has been done on distributed and parallel file systems. Although the motivations behind these two types of file systems are quite different, they share many underlying characteristics and techniques. Distributed file systems need to provide clients with efficient, usable, and scalable high-bandwidth access to stored data. Parallel file systems have a somewhat more straightforward goal: to allow multiple processes working together as a single task force to access a shared file or files in parallel. In these specialized systems, speed and scalability are the most important considerations, while issues like portability and ease of use are secondary. This paper will examine eight modern distributed and parallel file systems, looking at the design principles and goals behind each, their actual implementations, the advantages and disadvantages each entails, the innovations each system introduces, and the performance and scalability each offers. The paper concludes with a brief comparison of these systems.

The 9th International Conference on Advanced Communication Technology, 2007

Achieving high availability is a premium goal for many distributed file systems. File replication is a wellknown technique that is used to achieve this goal. It generally offers reduced client latencies and increases files availability. The objective of this paper is to achieve high availability of files through a novel intelligent replication method. To achieve this goal two algorithms are proposed, namely a primary replica assignment algorithm and an intelligent replica placement algorithms. The assignment algorithm uses an intelligent criteria based scheme to decide which one from a number of replicas to be assigned as the primary copy for a set of replicas. The placement algorithm also uses an intelligent criteria based scheme to decide on replication of files. The two algorithms have different criteria and the combined effect of both will facilitate for higher availability of files within the distributed file system.

2001 Eighteenth IEEE Symposium on Mass Storage Systems and Technologies, 2001

Consistent improvements in processor and memory technology have led to disks having greater processing power and cache memory in a compact size. This increased processing power and memory allows disks to execute more than just the basic disk operations, sometimes, even run user defined codes. Offloading part of the application processing to the disks can help reducing the latency of data manipulation as well as the amount of data transferred across the network. Such disks are called active disks. In this paper, a file system for the active-disk-based data server (ADFS) is proposed. All data files stored on active disks are provided with operations, forming objects. For some applications such as database, application-specific operations can be run by disk processors so that only the results are returned to clients, rather than whole data file is read by the clients. Therefore, ADFS is able to reduce the application-processing overhead of the system. In ADFS, part of file system functionality of the file manager can be delegated to active disks. Our implementation is based on CORBA specification. A set of file system interfaces is implemented for each module of each file system component. Part of the file system functionality, such as lookup, is embedded in the modules for active disks. We tested the performance of ADFS with 64 files in three-layer directory system. The performance results are obtained in a limited manner and presented in this paper. Our ADFS is basically stateful, but the delegation of file system functionality to disks allows our file system to be more scalable and helps to overcome some of the limitations of current prevailing file systems, like NFS, by greatly reducing the work of the central file manager. In addition, it is conceivable that the central file manager could be eliminated altogether.

ACM SIGOPS Operating Systems Review, 1985

This paper presents the design and rationale of a distributed file system for a network of more than 5000 personal computer workstations. While scale has been the dominant design influence, careful attention has also been paid to the goals of location transparency, user mobility and compatibility with existing operating system interfaces. Security is an important design consideration, and the mechanisms for it do not assume that the workstations or the network are secure. Caching of entire files at workstations is a key element in this design. A prototype of this system has been built and is in use by a user community of about 400 individuals. A refined implementation that will scale more gracefully and provide better performance is close to completion.

Advances in Computer and …, 2008

2009

The need and use of large scale distributed storage has rapidly increased in last few years. Organizations need Terabytes of storage for their operational data and backups. Large storage systems are the ultimate solution, but they are very expensive and require higher degree of skills for their operation and maintenance e.g. Storage Area Network (SAN). We propose "A Highly Scalable and Efficient Distributed File Storage System" that is reliable, inexpensive and easy to maintain. Our system is based on peer-to-peer network architecture. To ensure the reliability of the system we use a technique of erasure codes known as Luby Transform (LT). The System is designed for deployment in Local Area Networks (LAN) but with minimal changes it can be extended for Wide Area Networks (WAN) and Internet.

2001

Monolithic file servers are limited by the power of an individual system. Cluster file servers are limited by resource sharing and recovery issues as the number of cluster nodes increases. DiFFS is a file service architecture that allows system resources to be added (or removed) dynamically, e.g., storage and processors. Resources are partitioned in such a way that contention is avoided, while maintaining a single namespace. Resources may be heterogeneous, and geographically dispersed. This architecture has several advantages. A file's physical location is decoupled from its location in the namespace. This decoupling enables a powerful and flexible mechanism for the placement of file system objects. For example, different types of files, e.g., text or video, may reside anywhere in the namespace while being hosted by servers best suited to handling their content type. DiFFS also provides lightweight protocols for online dynamic reconfiguration (volume reassignment and object migration) to address fluctuating demand and potentially mobile file system entities. A DiFFS prototype has been implemented in Linux. Performance results indicate that the architecture achieves its flexibility and scalability goals without sacrificing performance.

Ibm Journal of Research and Development, 2008