Distributed Disk Store

MATEC Web of Conferences, 2018

Nowadays, a wide set of systems and application, especially in high performance computing, depends on distributed environments to process and analyses huge amounts of data. As we know, the amount of data increases enormously, and the goal to provide and develop efficient, scalable and reliable storage solutions has become one of the major issue for scientific computing. The storage solution used by big data systems is Distributed File Systems (DFSs), where DFS is used to build a hierarchical and unified view of multiple file servers and shares on the network. In this paper we will offer Hadoop Distributed File System (HDFS) as DFS in big data systems and we will present an Event-B as formal method that can be used in modeling, where Event-B is a mature formal method which has been widely used in a number of industry projects in a number of domains, such as automotive, transportation, space, business information, medical device and so on, And will propose using the Rodin as modeling tool for Event-B, which integrates modeling and proving as well as the Rodin platform is open source, so it supports a large number of plug-in tools.

Concurrency and Computation: Practice and Experience, 2008

The average PC now contains a large and increasing amount of storage with an ever greater amount left unused. We believe there is an opportunity for organizations to harness the vast unused storage capacity on their PCs to create a very large, low-cost, shared storage system. What is needed is the proper storage system architecture and software to exploit and manage the unused portions of existing PC storage devices across an organization and make it reliably accessible to users and applications. We call our vision of such a storage system Storage@desk (SD). This paper describes our first step towards the realization of SD-a study of machine and storage characteristics and usage in a model organization. We studied 729 PCs in an academic institution for 91 days, monitoring the configuration, load and usage of the major machine subsystems, i.e. disk, memory, CPU and network. To further analyze the availability characteristics of storage in an SD system, we performed a trace-driven simulation of some basic storage allocation strategies. This paper presents the results of our data collection efforts, our analysis of the data, our simulation results and our conclusion that an SD system is indeed feasible and holds promise as a cost-effective way to create massive storage systems.

Advances in Intelligent Systems and Computing, 2018

Nowadays, replication technique is widely used in data center storage systems to prevent data loss. Data popularity is a key factor in data replication as popular files are accessed most frequently and then they become unstable and unpredictable. Moreover, replicas placement is one of key issues that affect the performance of the system such as load balancing, data locality etc. Data locality is a fundamental problem to data-parallel applications that often happens and this problem leads to the decrease in performance. To address these challenges, this paper proposes a dynamic replication management scheme based on data popularity and data locality; it includes replica allocation and replica placement algorithms. Data locality, disk bandwidth, CPU processing speed and storage utilization are considered in the proposed data placement algorithm in order to achieve better data locality and load balancing effectively. Our proposed scheme will be effective for large-scale cloud storage.

The 9th International Conference on Advanced Communication Technology, 2007

Achieving high availability is a premium goal for many distributed file systems. File replication is a wellknown technique that is used to achieve this goal. It generally offers reduced client latencies and increases files availability. The objective of this paper is to achieve high availability of files through a novel intelligent replication method. To achieve this goal two algorithms are proposed, namely a primary replica assignment algorithm and an intelligent replica placement algorithms. The assignment algorithm uses an intelligent criteria based scheme to decide which one from a number of replicas to be assigned as the primary copy for a set of replicas. The placement algorithm also uses an intelligent criteria based scheme to decide on replication of files. The two algorithms have different criteria and the combined effect of both will facilitate for higher availability of files within the distributed file system.

The Journal of Supercomputing, 2010

Data replication techniques are widely used for improving availability in software applications. Replicated systems have traditionally assumed the fail-stop model, which limits fault tolerance. For this reason, there is a strong motivation to adopt the crash-recovery model, in which replicas can dynamically leave and join the system. With the aim to point out some key issues that must be considered when dealing with replication and recovery, we have implemented a replicated file server that satisfies the crash-recovery model, making use of a Group Communication System. According to our experiments, the most interesting results are that the type of replication and the number of replicas must be carefully determined, specially in update intensive scenarios; and, the variable overhead imposed by the recovery protocol to the system. From the latter, it would be convenient to adjust the desired trade-off between recovery time and system throughput in terms of the service state size and the number of missed operations.

International Journal of Computer Applications

Distributed File System (DFS) is acting as an extension to file system which manages files and data on multiple storage devices and provides more performance and reliability using various modern techniques. Outside world only sees the distributed file system as a single storage device and it is nothing but an interface to a great extent. In case of failure or heavy load very few Distributed file systems provide location transparency and redundancy to improve the data availability. Significant challenges for such a distributed file system are extended to a large number of storage nodes and providing reasonably. In this paper, there is been a few performance collaborative parameters have been studied viz. replication, fault tolerance and load balancing experimentally on various DFS.

2008

We start in 1.1 by citing some of the first attempts of implementing a distributed file system. In section 2 we present the most famous systems in the academic community, namely NFS, AFS, and SPRITE and some other experimental systems which brought important scientific contributions:

Distributed Operating Systems, 2010

The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications. Clients talk to a service that, in this case, happens to be a file service. The actual file service is offered by a single machine. Where multiple machines are used, the administrator is responsible for diving up file resources (e.g., home directories) among individual servers and the the entire resource (e.g., your home directory) is fully served from a single server. In Coda, which supports replication, it is up to the administrator to define the set of machines that hold volume replicas. This is not the kind of approach that we want to use if we want to divide our files among thousands or tens of thousands of machines. For an example of handling this environment, we will look at two closely-related file systems: the Google File System (GFS) and the Hadoop Distributed File System (HDFS). The latter is an open source version (and minor variant) of the former.

Concurrency and Computation: Practice and Experience, 2011

Large organizations always have a strong demand for storage from data-intensive applications and instruments. In this paper, we present the design, implementation, and evaluation of a new virtual storage system, Storage@desk, which can aggregate a large number of distributed machines within an organization to provide storage services with quality of service guarantees. Because storage virtualization is the prominent goal, Storage@desk provides clients with the abstraction of a hard drive by utilizing the Internet SCSI protocol. As such, data access to new storage services is transparent so that clients do not need to modify any existing applications nor change their current practices. Storage@desk replicates data and employs version-based journaling for high availability. It utilizes a market-based model for resource management and a feedback controller for automated performance control. We have developed a prototype of Storage@desk that implements the core components.

The issue of managing digital resources has become a major concern for educational and research institutions expanding with research laboratories, schools and departments. Some of these institutions have a major challenge as to how to manage huge quantities of educational resources required for research, teaching and learning. It is due to this challenge that this paper sought to design and implement a digital resource management system on the Hadoop Distributed File System (HDFS). The main model implemented in the development of the system was the incremental and iterative model. The development of the application took into consideration the user requirements and what the system sought to achieve after implementation. The system's architecture is divided into various levels and these are the user, systems management, data storage and infrastructural levels. The development of this application was focused on the systems management level. The system integration was achieved by connecting MYSQL database server with HDFS through the utilization of sqoop. Security features were implemented on the system to protect the system from attack. The system was tested to ensure that all the modules created were communicating perfectly and the system was producing the expected results.