Andrew File System

The BaBar experiment has accumulated many terabytes of data on particle physics reactions, accessed by a community of hundreds of users. Typical analysis tasks are C++ programs, individually written by the user, using shared templates and libraries. The resources have outgrown a single platform and a distributed computing model is needed. The grid provides the natural toolset. However, in contrast to the LHC experiments, BaBar has an existing user community with an existing non­Grid usage pattern, and providing users with an acceptable evolution presents a challenge. The ' Alibaba' system, developed as part of the UK GridPP project, provides the user with a familiar command line environment. It draws on the existing global file systems employed and understood by the current user base. The main difference is that they submit jobs with a 'gs ub' command that looks and feels like the familiar 'qs ub'. However it enables them to submit jobs to computer systems at...

1-3Department of CSE, Ramaiah Institute of Technology, Bengaluru, India -------------------------------------------------------------------------***---------------------------------------------------------------------AbstractDistributed system plays an important role in analysis and processing of huge amount of data. It is used for large number of applications such as forecasting, real time data analysis, and various other scientific application. Distributed File Systems has emerged as one of the most effective and convenient solutions to the issue of handling computations with increase in the quantity of data. Distributed file system (DFS) is mainly aimed to enable clients to use a common file system which paves the way for sharing data and resources over physically distributed machines. Many distributed computing projects have tried to implement and design distributed file systems with distinct architectures and have provided with various functionalities with them. This paper trie...

The amount of immutable files, such as images, video clips, audio files, and e-mail messages, is expected to grow significantly, as users actively generate, distribute, share, and re-use digital contents. In this paper, we present BeanFS, a distributed file system for a large number of immutable files. A key features of BeanFS include (1) the volume-based replication scheme to alleviate the metadata management overhead of the central metadata server, (2) a lightweight consistency maintenance protocol tailored to the simple access patterns of immutable files, and (3) volume synchronization and re-replication mechanisms which provide high availability against transient and permanent failures. BeanFS has been evaluated using a microbenchmark and a synthetic workload which reflects the characteristics of the real-world e-mail service system. Our measurement results indicate that the performance of BeanFS is scalable as the number of servers increases. In addition, volume synchronization and re-replication are quite effective in enhancing the availability of BeanFS.

1-3Department of CSE, Ramaiah Institute of Technology, Bengaluru, India -------------------------------------------------------------------------***---------------------------------------------------------------------AbstractDistributed system plays an important role in analysis and processing of huge amount of data. It is used for large number of applications such as forecasting, real time data analysis, and various other scientific application. Distributed File Systems has emerged as one of the most effective and convenient solutions to the issue of handling computations with increase in the quantity of data. Distributed file system (DFS) is mainly aimed to enable clients to use a common file system which paves the way for sharing data and resources over physically distributed machines. Many distributed computing projects have tried to implement and design distributed file systems with distinct architectures and have provided with various functionalities with them. This paper trie...

In preparation of the experiment at CERN's Large Hadron Collider (LHC), the ALICE collaboration has developed AliEn, a production environment that implements several components of the Grid paradigm needed to simulate, reconstruct and analyse data in a distributed way. Thanks to AliEn, the computing resources of a Virtual Organization can be seen and used as a single entity-any available node can execute jobs and access distributed datasets in a fully transparent way, wherever in the world a file or node might be. The system is built around Open Source components, uses the Web Services model and standard network protocols to implement the computing platform that is currently being used to produce and analyse Monte Carlo data at over 30 sites on four continents. Several other HEP experiments as well as medical projects (EU MammoGRID, INFN GP-CALMA) have expressed their interest in AliEn or some components of it. As progress is made in the definition of Grid standards and interoperability, our aim is to interface AliEn to emerging products from both Europe and the US. In particular, it is our intention to make AliEn services compatible with the Open Grid Services Architecture (OGSA). The aim of this paper is to present the current AliEn architecture and outline its future developments in the light of emerging standards.

Distributed File Systems are the backbone of how large volumes of data are stored. Hadoop File Systems, Google File Systems, and Network File Systems have all shifted the way data is maintained on servers. In terms of performance, fault tolerance, consistency, scalability, and availability, each file system has its own set of benefits and drawbacks. This presentation examines file system comparison research and suggests a criterion for selecting a certain file system. The presentation also looks into the pros and drawbacks of using a file system.

Distributed File Systems are the backbone of how large volumes of data are stored. Hadoop File Systems, Google File Systems, and Network File Systems have all shifted the way data is maintained on servers. In terms of performance, fault tolerance, consistency, scalability, and availability, each file system has its own set of benefits and drawbacks. This presentation examines file system comparison research and suggests a criterion for selecting a certain file system. The presentation also looks into the pros and drawbacks of using a file system.

Conclusions BdbServer++ has highlighted some of the areas of opportunity for the current middleware. This has been fed back into the development process for future releases. The issue of data location is however, one where ideal solutions remain a distant prospect. ...

This paper describes how the London e-Science Centre cluster MARS, a production 400+ Opteron CPU cluster, was integrated into the production Large Hadron Collider Compute Grid. It describes the practical issues that we encountered when deploying and maintaining this system, and details the techniques that were applied to resolve them.

We present two versions of a grid job submission system produced for the BaBar experiment. Both use globus job submission to process data spread across various sites, producing output which can be combined for analysis. The problems encountered with authorisation and authentication, data location, job submission, and the input and output sandboxes are described, as are the solutions. The total

Objective The objective of this document is to survey all available HEP use case documents, select from them the use cases which are common to Grid-like computing systems and identify the implications on the metadata catalogue which is to be used. For now, only physicist-level use cases are considered; that is, only the requirements of the end user. Furthermore, the term 'metadata' is taken to refer high-level information about jobs and collections of files (or 'datasets'). In particular, it does not include file and replica information. It is assumed that some lower-level catalogue will exit to perform this function, as laid out in the ARDA document [1].

This paper discusses the use of e-Science Grid in providing computational resources for modern international High Energy Physics (HEP) experiments. We investigate the suitability of the current generation of Grid software to provide the necessary resources to perform large-scale simulation of the experiment and analysis of data in the context of multinational collaboration.

The BABAR Collaboration, based at Stanford Linear Accelerator Center (SLAC), Stanford, US, has been performing physics reconstruction, simulation studies and data analysis for 8 years using a number of compute farms around the world. Recent developments in Grid technologies could provide a way to manage the distributed resources in a single coherent structure. We describe enhancements to the BABAR experiment's distributed skimmed dataset production system to make use of European Grid resources and present the results with regard to BABAR's latest cycle of skimmed dataset production. We compare the benefits of a local and Grid-based systems, the ease with which the system is managed and the challenges of integrating the Grid with legacy software. We compare job success rates and manageability issues between Grid and non-Grid production.

The BaBar experiment has been taking data since 1999. In 2001 the computing group started to evaluate the possibility to evolve toward a distributed computing model in a grid environment. We built a prototype system, based on the European Data Grid (EDG), to submit full-scale analysis and Monte Carlo simulation jobs. Computing elements, storage elements, and worker nodes have been installed at SLAC and at various European sites. A BaBar virtual organization (VO) and a test replica catalog (RC) are maintained in Manchester, U.K., and the experiment is using three EDG testbed resource brokers in the U.K. and in Italy. First analysis tests were performed under the assumption that a standard BaBar software release was available at the grid target sites, using RC to register information about the executable and the produced n-tuples. Hundreds of analysis jobs accessing either Objectivity or Root data files ran on the grid. We tested the Monte Carlo production using a farm of the INFN-grid testbed customized to install an Objectivity database and run BaBar simulation software. First simulation production tests were performed using standard Job Description Language commands and the output files were written on the closest storage element. A package that can be officially distributed to grid sites not specifically customized for BaBar has been prepared. We are studying the possibility to add a user friendly interface to access grid services for BaBar.

The BaBar experiment involves 500+ physicists spread across the world, with a requirement to access and analyse hundreds  of  Terabytes of data. Grid based tools are  being increasingly used for the manipulation of data and metadata, and for transferring data and analysis tasks between sites, facilitating and speeding up the analyses and making best use of equipment. We describe many of these developments: in authorisation,  job submission, and  data location.  The successes in introducing Grid technology as a solution to a pre-existent but overloaded computing model is described.

We have created a lightweight (server and client) intuitive command line interface through which users can submit jobs with complex software dependencies. We have added functionality that is specific to Babar jobs. gsub solves the input/output sandbox issues by the use of a global file system, currently AFS. Alibaba uses Gridsite to keep track of each job and display the status of sites in that grid.

gsub and Alibaba together provide a simple, easy-to-use grid submission and monitoring infrastructure that works and is currently being used by students and even professors for Particle Physics Analysis.

The BaBar experiment has accumulated many terabytes of data on particle physics reactions, accessed by a community of hundreds of users. Typical analysis tasks are C++ programs, individually written by the user, using shared templates and libraries. The resources have outgrown a single platform and a distributed computing model is needed. The grid provides the natural toolset. However, in contrast to the LHC experiments, BaBar has an existing user community with an existing non­Grid usage pattern, and providing users with an acceptable evolution presents a challenge.

The 'Alibaba'system, developed as part of the UK GridPP project, provides the user with a familiar command line environment. It draws on the existing global file systems employed and understood by the current user base. The main difference is that they submit jobs with a 'gsub'command that looks and feels like the familiar 'qsub'.However it enables them to submit jobs to computer systems at different institutions, with minimal requirements on the remote sites. Web based job monitoring is also provided. The problems and features (the input and output sandboxes, authentication, data location) and their solutions are described.