Papers by Colin Leavett-Brown
Proceedings of the 4th ACM workshop on Scientific cloud computing - Science Cloud '13, 2013
This paper describes the use of a distributed cloud computing system for high-throughput computin... more This paper describes the use of a distributed cloud computing system for high-throughput computing (HTC) scientific applications. The distributed cloud computing system is composed of a number of separate Infrastructure-as-a-Service (IaaS) clouds that are utilized in a unified infrastructure. The distributed cloud has been in production-quality operation for two years with approximately 500,000 completed jobs where a typical workload has 500 simultaneous embarrassinglyparallel jobs that run for approximately 12 hours. We review the design and implementation of the system which is based on pre-existing components and a number of custom components. We discuss the operation of the system, and describe our plans for the expansion to more sites and increased computing capacity.
HEP computing in a context-aware cloud environment
Proceedings of the 5th ACM workshop on Scientific cloud computing - ScienceCloud '14, 2014
ABSTRACT This paper describes the use of a distributed cloud computing system for high energy phy... more ABSTRACT This paper describes the use of a distributed cloud computing system for high energy physics (HEP) applications. The system is composed of IaaS clouds integrated into a unified infrastructure that has been in production for over two years. It continues to expand in scale and sites, encompassing more than twenty clouds on three continents. We are prototyping a new context-aware architecture that enables the virtual machines to make connections to both software and data repositories based on geolocation information. The new design will significantly enhance the ability of the system to scale to higher workloads and run data-intensive applications. We review the operation of the production system and describe our work towards a context-aware cloud system.
Journal of Physics: Conference Series, 2012
We show that distributed Infrastructure-as-a-Service (IaaS) compute clouds can be effectively use... more We show that distributed Infrastructure-as-a-Service (IaaS) compute clouds can be effectively used for the analysis of high energy physics data. We have designed a distributed cloud system that works with any application using large input data sets requiring a high throughput computing environment. The system uses IaaS-enabled science and commercial clusters in Canada and the United States. We describe the process in which a user prepares an analysis virtual machine (VM) and submits batch jobs to a central scheduler. The system boots the user-specific VM on one of the IaaS clouds, runs the jobs and returns the output to the user. The user application accesses a central database for calibration data during the execution of the application. Similarly, the data is located in a central location and streamed by the running application. The system can easily run one hundred simultaneous jobs in an efficient manner and should scale to many hundreds and possibly thousands of user jobs.
Journal of Physics: Conference Series, 2010
The recent increase in availability of Infrastructure-as-a-Service (IaaS) computing clouds provid... more The recent increase in availability of Infrastructure-as-a-Service (IaaS) computing clouds provides a new way for researchers to run complex scientific applications. However, using cloud resources for a large number of research jobs requires significant effort and expertise. Furthermore, running jobs on many different clouds presents even more difficulty. In order to make it easy for researchers to deploy scientific applications across many cloud resources, we have developed a virtual machine resource manager (Cloud Scheduler) for distributed compute clouds. In response to a user's job submission to a batch system, the Cloud Scheduler manages the distribution and deployment of user-customized virtual machines across multiple clouds. We describe the motivation for and implementation of a distributed cloud using the Cloud Scheduler that is spread across both commercial and dedicated private sites, and present some early results of scientific data analysis using the system.
dCache with tape storage for High Energy Physics applications
Journal of Physics: Conference Series, 2010
An interface between dCache and the local Tivoli Storage Manager (TSM) tape storage facility has ... more An interface between dCache and the local Tivoli Storage Manager (TSM) tape storage facility has been developed at the University of Victoria (UVic) for High Energy Physics (HEP) applications. The interface is responsible for transferring the data from disk pools to tape and retrieving data from tape to disk pools. It also checks the consistency between the PNFS filename space
The availability of Infrastructure-as-a-Service (IaaS) computing clouds gives researchers access ... more The availability of Infrastructure-as-a-Service (IaaS) computing clouds gives researchers access to a large set of new resources for running complex scientific applications. However, exploiting cloud resources for large numbers of jobs requires significant effort and expertise. In order to make it simple and transparent for researchers to deploy their applications, we have developed a virtual machine resource manager (Cloud Scheduler) for distributed compute clouds. Cloud Scheduler boots and manages the user-customized virtual machines in response to a user's job submission. We describe the motivation and design of the Cloud Scheduler and present results on its use on both science and commercial clouds.
Uploads
Papers by Colin Leavett-Brown