Academia.eduAcademia.edu

Figure 6 – uploaded by Scott Callaghan

See full PDFdownloadDownload figure
The main concept in tl he new data management approach is to be able to place the data staging element anywhere in the execution environment. I ample, the data storage e n the previous Open Science Grid ex- ement was an SRM/GridFTP/iRODS server placed near the compute resources. Let’s consider the same workflow, but this time running on an HPC cluster with a shared filesystem (Figure oh. Figure 6: Data management on a shared filesystem resource (only the remote execution environment is shown).

Figure 6 The main concept in tl he new data management approach is to be able to place the data staging element anywhere in the execution environment. I ample, the data storage e n the previous Open Science Grid ex- ement was an SRM/GridFTP/iRODS server placed near the compute resources. Let’s consider the same workflow, but this time running on an HPC cluster with a shared filesystem (Figure oh. Figure 6: Data management on a shared filesystem resource (only the remote execution environment is shown).

Related Figures (15)

Figure 1: Pegasus WMS Architecture.
Figure 1: Pegasus WMS Architecture.
Figure 3: Hello World DAX (Directed Acyclic graph in XML). Figure 2: Python DAX generator code for Hello World DAX.
Figure 3: Hello World DAX (Directed Acyclic graph in XML). Figure 2: Python DAX generator code for Hello World DAX.
Figure 4: Translation of a simple Hello World DAX to an Executable Workflow
Figure 4: Translation of a simple Hello World DAX to an Executable Workflow
Figure 5: Data flow for a Pegasus Workflows planned with non-shared filesystem mode, running on the Open Science Grid.
Figure 5: Data flow for a Pegasus Workflows planned with non-shared filesystem mode, running on the Open Science Grid.
4.3. Task Clustering
4.3. Task Clustering
Figure 9: Example of level-based clustering. Figure 10: Example of label-based clustering.
Figure 9: Example of level-based clustering. Figure 10: Example of label-based clustering.
Figure 8: Translation from Abstract to Executable Workflow.
Figure 8: Translation from Abstract to Executable Workflow.
Figure 11: Hierarchical Workflows. Expansion ends when a DAX with only compute jobs is encountered.
Figure 11: Hierarchical Workflows. Expansion ends when a DAX with only compute jobs is encountered.
Figure 12: The Galactic Plane workflow, showing the top-level workflow with 1,001 data find tasks and sub-workflows.
Figure 12: The Galactic Plane workflow, showing the top-level workflow with 1,001 data find tasks and sub-workflows.
Figure 14: Workflow Gantt Chart and Job Distribution Pie Charts displayed in pegasus-dashboard. Figure 13: Workflow Statistics displayed in pegasus-dashboard.
Figure 14: Workflow Gantt Chart and Job Distribution Pie Charts displayed in pegasus-dashboard. Figure 13: Workflow Statistics displayed in pegasus-dashboard.
Pegasus also includes a lightweight web dashboard t hat en- ables easy monitoring and online exploration of workflows based on an embedded board can display all the workflows (running and com: for a particular user. It allows users to browse their flows. For each workflow, users can see a summary workflow run including runtime statistics (snapshot dis in Figure|13 Figure[14p illustratin time and pie charts d web server written in Python. The dash- pleted) work- of the played such as the number of jobs completed, failed, and workflow walltime, and can browse data collected for each job by the system. The dashboard can also generate different types of charts such as workflow gantt charts (snapshot displayed in sis of count and runtime. g the the progress of the workflow through epicting the distribution of jobs on the ba-
Pegasus also includes a lightweight web dashboard t hat en- ables easy monitoring and online exploration of workflows based on an embedded board can display all the workflows (running and com: for a particular user. It allows users to browse their flows. For each workflow, users can see a summary workflow run including runtime statistics (snapshot dis in Figure|13 Figure[14p illustratin time and pie charts d web server written in Python. The dash- pleted) work- of the played such as the number of jobs completed, failed, and workflow walltime, and can browse data collected for each job by the system. The dashboard can also generate different types of charts such as workflow gantt charts (snapshot displayed in sis of count and runtime. g the the progress of the workflow through epicting the distribution of jobs on the ba-
Table 1: Comparison of CyberShake Studies
Table 1: Comparison of CyberShake Studies
Figure 16: Cybershake Hierarchical Workflow.
Figure 16: Cybershake Hierarchical Workflow.
Related topics:
Information SystemsDistributed ComputingScientific WorkflowsWorkflow Management SystemsWorkflow Management System
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved