Data Management for the RedisDG Scientific Workflow Engine

2009 Congress on Services - I, 2009

With the advent of Grid and service-oriented technologies, scientific workflows have been introduced in response to the increasing demand of researchers for assembling diverse, highly-specialized applications, allowing them to exchange large heterogeneous datasets in order to accomplish a complex scientific task. Much research has already been done to provide efficient scientific workflow management systems (WfMS). However, most of such WfMS are coordinating and executing workflows in a centralized fashion. This creates a single point of failure, forms a scalability bottleneck, and often leads to excessive traffic routed back to the coordinator. Additionally, none of the available WfMS provides means for dynamic data transformation between services in order to overcome the data heterogeneity problem. This work presents a new approach for scientific workflow management targeted to provide ways for an efficient distributed execution of data-intensive workflows. The proposed approach reduces the communication traffic between services and overcomes the data heterogeneity problem. Moreover, it allows full control over long-running applications, as well as provides support for smart re-run, distributed fault handling and distributed load balancing.

A scientific workflow management system can be considered as a binding agent which brings together scientists and distributed resources. A workflow graph plays the central role in such a system as it is the component understood by both scientist and machine. Making sense out of a scientific workflow graphs is undoubtedly, the first and foremost responsibility of a workflow management system. Typical systems include an orchestration engine which models a workflow and schedules individual components onto distributed resources. As part of the WS-VLAM, we present an alternative orchestration engine which takes a different stand on interpreting the workflow graph. Whilst the current engine in WS-VLAM models the graph as a process network where components are tightly coupled through communication channels, the Datafluo engine models the graph as a dataflow network with farming capabilities. In this dissertation, we present the Datafluo architecture followed by a prototype implementation. The prototype is taken through its passes using scientific workflow applications where the generated results demonstrate the orchestration features. Through our results, we show how dataflow techniques reduce queue waiting times whilst farming techniques circumvent common workflow bottlenecks.

HAL (Le Centre pour la Communication Scientifique Directe), 2017

Scientific workflows emerged as a technology that enables scientists to undertake computational scientific experiments. Workflow enactors map workflow tasks onto distributed resources, hiding the inherent complexity of distributed infrastructures to the users. In the past, while the emphasis has been focused in adapting the workflow structure onto the resources, today the emergence of the cloud computing paradigm enables us to adapt the resources to the workflow tasks based on their characteristics. In this paper, we examine the concept of Cloud Scientific Workflow, and propose a new architectural approach based on autonomic principles, guided by a combination of high-level and low-level policies. High-level policies enable the workflow enactor to choose among a number of workflow structure transformations that better suit the underlying resources dynamically depending on the context, whereas low-level policies enable the autonomic resource manager to adjust the required computational power to the workload derived from a scientific workflow specification, exploiting the cloud elasticity property, and to cope with performance fluctuations or unexpected events in the cloud infrastructure. The novelty of our approach is the combination of both policies that can lead to higher degrees of dynamism. The key enabling architectural components for such a dynamism are the petri-net based performance models for implementing high-level policies and MOSt-CB system for the adaptation to multi-cloud environments.

IEEE Transactions on Services Computing, 2014

Cloud computing is an emerging computing paradigm that can offer unprecedented scalability and resources on demand, and is getting more and more adoption in the science community, while scientific workflow management systems provide essential support such as management of data and task dependencies, job scheduling and execution, provenance tracking, etc., to scientific computing. As we are entering into a "big data" era, it is imperative to migrate scientific workflow management systems into the Cloud to manage the ever increasing data scale and analysis complexity. We propose a reference service framework for integrating scientific workflow management systems into various Cloud platforms, which consists of eight major components, including Cloud workflow management service, Cloud resource manager, etc., and 6 interfaces between them. We also present a reference framework for the implementation of Cloud Resource Manager, which is responsible for the provisioning and management of virtual resources in the Cloud. We discuss our implementation of the framework by integrating the Swift scientific workflow management system with the OpenNebula and Eucalyptus Cloud platforms, and demonstrate the capability of the solution using a NASA MODIS image processing workflow and a production deployment on the Science@Guoshi network with support for the Montage image mosaic workflow.

Scalable Computing: Practice and Experience

Workflows management systems (WfMS) are aimed for designing, scheduling, executing, reusing, and sharing workflows in distributed environments like the Cloud computing. With the emergence of e-science workflows, which are used in different domains like astronomy, life science, and physics, to model and execute vast series of dependents functionalities and a large amount of manipulated data, the workflow management systems are required to provide customizable programming environments to ease the programming effort required by scientists to orchestrate a computational science experiment. A key issue for e-science WfMS is how to deal with the change of the execution environment constraints and the variability and confliction of end users and cloud providers objectives for the execution of the same workflow or sub-workflow. They have to customize their management processes to insure the adaptability of the execution environment to the scientific workflows specificities, especially when dealing with large-scale (data, computing, I/O)-intensive workflows. In this paper, we propose a dynamically re-configurable framework for the deployment of scientific workflows in the Cloud (called DR-SWDF) that allows customizing the workflow deployment process according to a set of objectives and constraints of end users or cloud providers defined differently for the tasks or partitions of the same workflow. The DR-SWDF framework offers a K-means based algorithm that allows dynamically clustering the input workflows or sub-workflows in order to identify the most convenient techniques or algorithms to be applied for their scheduling and deployment in the cloud. The simulations results run on three examples large-scale scientific workflows show that our proposed framework can achieve better results than the use of a generic purpose approach.

ArXiv, 2021

Scientific workflows are a cornerstone of modern scientific computing, and they have underpinned some of the most significant discoveries of the last decade. Many of these workflows have high computational, storage, and/or communication demands, and thus must execute on a wide range of large-scale platforms, from large clouds to upcoming exascale HPC platforms. Workflows will play a crucial role in the data-oriented and post-Moore's computing landscape as they democratize the application of cutting-edge research techniques, computationally intensive methods, and use of new computing platforms. As workflows continue to be adopted by scientific projects and user communities, they are becoming more complex. Workflows are increasingly composed of tasks that perform computations such as short machine learning inference, multi-node simulations, long-running machine learning model training, amongst others, and thus increasingly rely on heterogeneous architectures that include CPUs but ...

As scientific applications generate and consume data at ever-increasing rates, scientific workflow systems that manage the growing complexity of analyses and data movement will increase in importance. The goal of our work is to improve the overall performance of scientific workflows by using policy to improve data staging into and out of computational resources. We developed a Policy Service that gives advice to the workflow system about how to stage data, including advice on the order of data transfers and on transfer parameters. The Policy Service gives this advice based on its knowledge of ongoing transfers, recent transfer performance, and the current allocation of resources for data staging. The paper describes the architecture of the Policy Service and its integration with the Pegasus Workflow Management System. It employs a range of policies for data staging, and presents performance results for one policy that does a greedy allocation of data transfer streams between source and destination sites. The results show performance improvements for a data-intensive workflow: the Montage astronomy workflow augmented to perform additional large data staging operations.

12th International Symposium …, 2010

IEEE Access, 2021

Cloud computing provides solutions to a large number of organizations in terms of hosting systems and services. The services provided by cloud computing are broadly used for business and scientific applications. Business applications are task oriented applications and structured into business workflows. Whereas, scientific applications are data oriented and compute intensive applications and structured into scientific workflows. Scientific workflows are managed through scientific workflows management and scheduling systems. Recently, a significant amount of research is carried out on management and scheduling of scientific workflow applications. This study presents a comprehensive review on scientific workflows management and scheduling in cloud computing. It provides an overview of existing surveys on scientific workflows management systems. It presents a taxonomy of scientific workflow applications and characteristics. It shows the working of existing scientific workflows management and scheduling techniques including resource scheduling, fault-tolerant scheduling and energy efficient scheduling. It provides discussion on various performance evaluation parameters along with definition and equation. It also provides discussion on various performance evaluation platforms used for evaluation of scientific workflows management and scheduling strategies. It finds evaluation platforms used for the evaluation of scientific workflows techniques based on various performance evaluation parameters. It also finds various design goals for presenting new scientific workflow management techniques. Finally, it explores the open research issues that require attention and high importance.

Lecture Notes in Computer …, 2004

Grid workflows can be seen as special scientific workflows involving high performance and/or high throughput computational tasks. Much work in grid workflows has focused on improving application performance through schedulers that optimize the use of computational resources and bandwidth. As high-end computing resources are becoming more of a commodity that is available to new scientific communities, there is an increasing need to also improve the design and reusability "performance" of scientific workflow systems. To this end, we are developing a framework that supports the design and reuse of grid workflows. Individual workflow components (e.g., for data movement, database querying, job scheduling, remote execution etc.) are abstracted into a set of generic, reusable tasks. Instantiations of these common tasks can be functionally equivalent atomic components (called actors) or composite components (so-called composite actors or subworkflows). In this way, a grid workflow designer does not have to commit to a particular Grid technology when developing a scientific workflow; instead different technologies (e.g. GridFTP, SRB, and scp) can be used interchangeably and in concert. We illustrate the application of our framework using two real-world Grid workflows from different scientific domains, i.e., cheminformatics and bioinformatics, respectively.