Towards a Grid Platform for Scientific Workflows Management

Scientific workflows are used in a number of different scientific domains. The interest for using workflows grows because of the progress of Grid computational techniques and the increased resources availability. Moreover, there are a number of scientific workflow systems that allow building workflows needed for different scientific applications and scientific experiments. In this paper, we analyze the characteristics of two open source scientific workflow systems: OpenMole and Taverna. Both systems are designed to provide environments for parallel execution of processes, to support graphical workflow design, to offer scalability and remote execution of workflows (grids, clusters, clouds). The former is relatively new, but both are open to users if they want to contribute to their development. Although these two systems have a number of common features, they differ in the way of workflow composition, execution and implementation logic. In order to show and verify the similarities an...

12th International Symposium …, 2010

A scientific workflow management system can be considered as a binding agent which brings together scientists and distributed resources. A workflow graph plays the central role in such a system as it is the component understood by both scientist and machine. Making sense out of a scientific workflow graphs is undoubtedly, the first and foremost responsibility of a workflow management system. Typical systems include an orchestration engine which models a workflow and schedules individual components onto distributed resources. As part of the WS-VLAM, we present an alternative orchestration engine which takes a different stand on interpreting the workflow graph. Whilst the current engine in WS-VLAM models the graph as a process network where components are tightly coupled through communication channels, the Datafluo engine models the graph as a dataflow network with farming capabilities. In this dissertation, we present the Datafluo architecture followed by a prototype implementation. The prototype is taken through its passes using scientific workflow applications where the generated results demonstrate the orchestration features. Through our results, we show how dataflow techniques reduce queue waiting times whilst farming techniques circumvent common workflow bottlenecks.

ArXiv, 2021

Scientific workflows are a cornerstone of modern scientific computing, and they have underpinned some of the most significant discoveries of the last decade. Many of these workflows have high computational, storage, and/or communication demands, and thus must execute on a wide range of large-scale platforms, from large clouds to upcoming exascale HPC platforms. Workflows will play a crucial role in the data-oriented and post-Moore's computing landscape as they democratize the application of cutting-edge research techniques, computationally intensive methods, and use of new computing platforms. As workflows continue to be adopted by scientific projects and user communities, they are becoming more complex. Workflows are increasingly composed of tasks that perform computations such as short machine learning inference, multi-node simulations, long-running machine learning model training, amongst others, and thus increasingly rely on heterogeneous architectures that include CPUs but ...

2005

It is a complex task to design and implement a workflow management system that supports scalable executions of large-scale scientific workflows for dynamic and heterogeneous Grid environments. In this paper we describe the Distributed workflow Enactment Engine (DEE) of the ASKALON Grid application development environment for Grid computing. DEE proposes a de-centralized architecture that simplifies and reduces the overhead for managing large workflows through partitioning, improved data locality, and reduced workflow-level checkpointing overhead. We report experimental results for a real-world material science workflow application.

Journal of Grid Computing, 2013

Today there exist a wide variety of scientific workflow management systems, each designed to fulfill the needs of a certain scientific community. Unfortunately, once a workflow application has been designed in one particular system it becomes very hard to share it with users working with different systems. Portability of workflows and interoperability between current systems barely exists. In this work, we present the fine-grained interoperability solution proposed in the SHIWA European project that brings together four representative European workflow systems: ASKALON, MOTEUR, WS-PGRADE, and Triana. The proposed interoperability is realised at two levels of abstraction: abstract and concrete. At the abstract level, we propose a generic Interoperable Workflow Intermediate Representation (IWIR) that can be used as a common bridge for translating workflows between different languages independent of the underlying distributed computing infrastructure. At the concrete level, we propose a bundling technique that aggregates the abstract IWIR representation and concrete task representations to enable workflow instantiation, execution and scheduling. We illustrate case studies using two real-workflow applications designed in a native environment and then translated and executed by a foreign workflow system in a foreign distributed computing infrastructure.

the Workflow in Grid …, 2004

Concurrency and Computation: Practice and Experience, 2009

2011

Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07), 2007

This paper presents a fault tolerance framework for applications that process data using a distributed network of user-defined operations in a pipelined fashion. The framework saves intermediate results and messages exchanged among application components in a distributed data management system to facilitate quick recovery from failures. The experimental results show that the framework scales well and our approach introduces very little overhead to application execution.