Recovery Protocols For Flash File Systems

Proceedings of the …, 2003

File system consistency frequently involves a choice between raw performance and integrity guarantees. A few software-based solutions for this problem have appeared and are currently being used on some commercial operating systems; these include log-structured file systems, journaling file systems, and soft updates. In this paper, we propose meta-data snapshotting as a low-cost, scalable, and simple mechanism that provides file system integrity. It allows the safe use of write-back caching by making successive snapshots of the meta-data using copy-onwrite, and atomically committing the snapshot to stable storage without interrupting file system availability. In the presence of system failures, no file system checker or any other operation is necessary to mount the file system, therefore it greatly improves system availability. This paper describes meta-data snapshotting, and its incorporation into a file system available for the Linux and K42 operating systems. We show that metadata snapshotting has low overhead: for a microbenchmark, and two macrobenchmarks, the measured overhead is of at most 4%, when compared to a completely asynchronous file system, with no consistency guarantees. Our experiments also show that it induces less overhead then a write-ahead journaling file system, and it scales much better when the number of clients and file system operations grows.

"With the advent of large and fast storage devices (eg. SSDs, Flash Disks) file system sizes have grown beyond bounds. Large volume of dynamically changing data makes it difficult to store data on the fly. Consistent backup ensuring integrity and correctness of data is of utmost important. Offline backup schemes result in heavy losses due to large down time. Online backup schemes offer weak consistency guarantees. Transactional file systems offer strong guarantees; however, they hinder the normal functioning of user applications. This results in transaction aborts and sub-optimal performance. This work proposes an online backup approach that circumvents user transaction aborts due to the backup process in a transactional file system. Conventional serialization approaches ensure consistency with transaction aborts. This work defines a concept “Conflict Dependence” that helps identify conflicts within backup and user transactions. An efficient implementation within a log-structured file system (LFS) is proposed. It does not require aborts because of the backup process. The backup scheme extends naturally to a versioning in a LFS. "

1992

Robotic storage devices offer huge storage capacity at a low cost per byte, but with large access times. Integrating these devices into the storage hierarchy presents a challenge to file system designers. Log-structured file systems (LFSs) were developed to reduce latencies involved in accessing disk devices, but their sequential write patterns match well with tertiary storage characteristics. Unfortunately, existing versions only manage memory caches and disks, and do not support a broader storage hierarchy.

2004

File systems are responsible for storing data efficiently and reliably. Databases also have similar tasks, but generally have more concern for reliability than file systems. We have developed a file system, KBDBFS, on top of an in-kernel database. KBDBFS supports standard file system operations, as well as other useful extensions such as extended attributes, ACLs, and a new operation to retrieve all the names of a file with hardlinks. To implement KBDBFS, we have ported the Berkeley DB software into the Linux kernel. Berkeley DB provides efficient and persistent storage of key-value pairs using hash tables and Btrees with full ACID semantics. Berkeley DB is a stable, high-quality, fully-featured, and widelyused embedded database. KBDBFS's fundamental operations manipulate a database schema, rather than low-level objects like bitmaps and allocation tables, so it is easily extensible. KBDBFS uses BDB transactions around its file system operations; this ensures that KBDBFS operations are reliable. KBDBFS also exports the transaction API to user-level processes, allowing them to perform several file system operations atomically. Our performance evaluation shows that KBDBFS has acceptable overheads for user-like workloads.

USENIX Annual Technical Conference, 2018

We introduce TxFS, a novel transactional file system that builds upon a file system's atomic-update mechanism such as journaling. Though prior work has explored a number of transactional file systems, TxFS has a unique set of properties: a simple API, portability across different hardware, high performance, low complexity (by building on the journal), and full ACID transactions. We port SQLite and Git to use TxFS, and experimentally show that TxFS provides strong crash consistency while providing equal or better performance.

1997

We present the design for a fault-tolerant persistent data storage in a wide-area distributed cooperative engineering environment. We discuss some of the current options in file system design and, in particular, the relation between file mobility and transaction protocols. We believe that file mobility, and consequently coherence protocols, global file naming and versioning, are issues whose exploration and use will lead to improved transaction protocols and to an increased applicability of transactional file systems.

Operating Systems Review, 2007

In many workloads, most write operations performed on a file system modify only a small number of blocks. The logstructured file system was designed for such a workload, additionally with the aim of fast crash recovery and system snapshots. Surprisingly, although implemented for Berkeley Sprite and BSD systems, there was no complete implementation for the current Linux kernel. In this paper, we present a complete implementation of the log-structured file system for the Linux kernel, which includes a user-space garbage collector and additional tools. We evaluate the measurements obtained in several test cases and compare the results with widely-used ext3.

1979

An algorithm is described which guarantees reliable storage of data in a distributed system, even when different portions of the data base, stored on separate machines, are updated as part of a single transaction. The algorithm is implemented by a hierarchy of rather simple abstractions, and it works properly regardless of crashes of the client or servers. Some care is taken to state precisely the assumptions about the physical components of the system (storage, processors and communication).

Design Automation for Embedded Systems, 2010

Since flash memory has many attractive characteristics such as high performance, non-volatility, low power consumption and shock resistance, it has been widely used as a storage media in embedded and computer system environments. However, there are many shortcomings in flash memory such as potentially high I/O latency due to erase-before-write and poor durability due to limited erase cycles. To address these performance and reliability anomalies, many large-scale storage systems use redundancy-based parallel access schemes such as RAID techniques. However, such redundancy-based schemes incur high overhead due to generating and storing redundancy information, especially in flash-based storage systems. In this paper, we propose a novel and performance-effective approach using a redundancy-based data management scheme in flash storage, called Flash-aware Redundancy Array. The proposed technique not only reduces the redundancy management overhead by performing redundancy update operations during idle periods, but also provides a preventive mechanism to recover data from unexpected read errors occurring before such redundancy update operations finish. From the experiments, we found that the proposed technique improves flash-based storage systems by 19% in average execution time as compared to other redundancy-based approaches.

Procedia Technology, 2014

Data recovery is a significant problem that presents a real challenge to forensics investigators today. File carvers have traditionally helped mitigate these difficulties. However, two issues still present significant challenges-1) Prior knowledge of file types is required for building file carvers, and 2) fragmentation prevents file carvers from successful recovery. In previous research, we proposed a framework for recovering deleted files without prior knowledge of file types and with the existence of fragmentation. In this paper, we introduce the design and a functioning implementation of our system by modifying an exFat filesystem running on top of FUSE. Evaluation of the overhead of our filesystem shows only a 5% decrease in performance in write operations when compared to an unmodified exFat filesystem, and almost identical read measurements. Our system also shows significantly better recovery rates in the presence of fragmentation when compared to two selected file carvers.