File Synchronization

This With the advance of Web Services technologies and the emergence of Web Services into the information space, tremendous opportunities for empowering users and organizations appear in various application domains including electronic commerce, travel, intelligence information gathering and analysis, health care, digital government, etc. In fact, Web services appear to be s solution for integrating distributed, autonomous and heterogeneous information sources. However, as Web services evolve in a dynamic environment which is the Internet many changes can occur and affect them. A Web service is affected when one or more of its associated information sources is affected by schema changes. Changes can alter the information sources contents but also their schemas which may render Web services partially or totally undefined. In this paper, we propose a solution for integrating information sources into Web services. Then we tackle the Web service synchronization problem by substituting the affected information sources. Our work is illustrated with a healthcare case study.

Remote Differential Compression (RDC) protocols can efficiently update files over a limitedbandwidth network when two sites have roughly similar files; no site needs to know the content of another's files a priori. We present a heuristic approach to identify and transfer the file differences that is based on finding similar files, subdividing the files into chunks, and comparing chunk signatures. Our work significantly improves upon previous protocols such as LBFS and RSYNC in three ways. Firstly, we present a novel algorithm to efficiently find the client files that are the most similar to a given server file. Our algorithm requires 96 bits of meta-data per file, independent of file size, and thus allows us to keep the metadata in memory and eliminate the need for expensive disk seeks. Secondly, we show that RDC can be applied recursively to signatures to reduce the transfer cost for large files. Thirdly, we describe new ways to subdivide files into chunks that identify file differences more accurately. We have implemented our approach in DFSR, a state-based multimaster file replication service shipping as part of Windows Server 2003 R2. Our experimental results show that similarity detection produces results comparable to LBFS while incurring a much smaller overhead for maintaining the metadata. Recursive signature transfer further increases replication efficiency by up to several orders of magnitude.

Remote Differential Compression (RDC) protocols can efficiently update files over a limitedbandwidth network when two sites have roughly similar files; no site needs to know the content of another's files a priori. We present a heuristic approach to identify and transfer the file differences that is based on finding similar files, subdividing the files into chunks, and comparing chunk signatures. Our work significantly improves upon previous protocols such as LBFS and RSYNC in three ways. Firstly, we present a novel algorithm to efficiently find the client files that are the most similar to a given server file. Our algorithm requires 96 bits of meta-data per file, independent of file size, and thus allows us to keep the metadata in memory and eliminate the need for expensive disk seeks. Secondly, we show that RDC can be applied recursively to signatures to reduce the transfer cost for large files. Thirdly, we describe new ways to subdivide files into chunks that identify file differences more accurately. We have implemented our approach in DFSR, a state-based multimaster file replication service shipping as part of Windows Server 2003 R2. Our experimental results show that similarity detection produces results comparable to LBFS while incurring a much smaller overhead for maintaining the metadata. Recursive signature transfer further increases replication efficiency by up to several orders of magnitude.

We consider the problem of data reconciliation, which we model as two separate multisets of data that must be reconciled with minimum communication. Under this model, we show that the problem of reconciliation is equivalent to a variant of the graph coloring problem and provide consequent upper and lower bounds on the communication complexity of reconciliation.

Remote Differential Compression (RDC) protocols can efficiently update files over a limitedbandwidth network when two sites have roughly similar files; no site needs to know the content of another's files a priori. We present a heuristic approach to identify and transfer the file differences that is based on finding similar files, subdividing the files into chunks, and comparing chunk signatures. Our work significantly improves upon previous protocols such as LBFS and RSYNC in three ways. Firstly, we present a novel algorithm to efficiently find the client files that are the most similar to a given server file. Our algorithm requires 96 bits of meta-data per file, independent of file size, and thus allows us to keep the metadata in memory and eliminate the need for expensive disk seeks. Secondly, we show that RDC can be applied recursively to signatures to reduce the transfer cost for large files. Thirdly, we describe new ways to subdivide files into chunks that identify file differences more accurately. We have implemented our approach in DFSR, a state-based multimaster file replication service shipping as part of Windows Server 2003 R2. Our experimental results show that similarity detection produces results comparable to LBFS while incurring a much smaller overhead for maintaining the metadata. Recursive signature transfer further increases replication efficiency by up to several orders of magnitude.

this paper. We examine the data reconciliation problem within a generalized framework in which differences between multi-sets correspond to evaluations of arbitrary error functions. We limit communication between reconciling hosts to a single message and, under such conditions, the problem of data reconciliation is equivalent to a variation of the problem of graph coloring: second-order coloring or distance-2 coloring, from

Differential compression (aka, delta encoding) is a special category for data de-duplication.  It can find many applications in various domains such as data backup, software revision control  systems, software incremental update, file synchronization over network, to name just a few. This talk will introduce a taxonomy of how to categorize delta encoding schemes in various applications. Pros & cons of each scheme will be investigated in depth.

Mathematical modeling is an important step for developing many advanced technologies in various domains such as network security, data mining and etc…  This lecture introduces a process  that the speaker summarizes from his past practice of mathematical modeling and  algorithmic solutions in IT industry, as  an applied mathematician, algorithm specialist or software engineer , and even as an  entrepreneur.  A practical problem from DLP  system will be used as an example for creating math models and providing algorithmic solutions.

Abstract-EnduraData's EDpCloud is a host based high performance replication software solution that allows the enterprise to replicate data and synchronize files between heterogeneous operating systems and architectures over LANs, WANs, WLANs or slow networks. The software takes advantage of parallel disk I/O, parallel processing, dynamic compression and efficient network resource utilization to achieve high throughput. The software has several modules that work in parallel and in tandem to deliver greater performance, higher reliability and better security. EDpCloud uses journaling and multithreading to optimize data distribution and aggregation reliably and efficiently between geographically dispersed systems. This paper provides a brief overview of how EDpCloud works.