Please, visit our new project website:

Panorama 360: Performance Data Capture and Analysis for End-to-end Scientific Workflows

Diamonds that deliver!
Neutrons, simulation analysis of tRNA-nanodiamond combo could transform drug delivery design principles.

A unique combination of experimentation and simulation was used to shed light on the design principles for improved delivery of RNA drugs, which are promising candidates in the treatment of a number of medical conditions including cancers and genetic disorders.

Nanodiamonds are good delivery candidates due to their spherical shape, biocompatibility and low toxicity. And because their surfaces can be easily tailored to facilitate the attachment of various medicinal molecules, nanodiamonds have tremendous potential for the delivery of a vast range of therapies.

The discovery involved ORNL’s Spallation Neutron Source, which provides the most intense pulsed neutron beams in the world for scientific research and industrial development, and ORNL’s Titan supercomputer, the nation’s most powerful for open science. By comparing the SNS neutron scattering data with the data from the team’s molecular dynamics simulations on Titan, the researchers have confirmed that nanodiamonds enhance the dynamics of tRNA when in the presence of water; This analysis was performed with the Pegasus WMS.

The DOE Panorama project has developed an SNS workflow to confirm that nanodiamonds enhance the dynamics of tRNA in presence of water. The workflow, enacted by the Pegasus Workflow Management System, calculates the epsilon that best matches experimental data. These calculations were for 10 ns each and the workflows used almost 400,000 CPU hours of time on DOE leadership class systems.

Image by Michael Mattheson, OLCF, ORNL

Panorama project was awarded Best Demo Second Runner-up at GENI Engineering Conference 25

Data Flow Prioritization for Scientific Workflows Using A Virtual SDX on ExoGENI

Anirban Mandal, Paul Ruth, Ilya Baldin, Rafael Ferreira da Silva, Ewa Deelman

Abstract. We presented a novel, dynamically adaptable networked cloud infrastructure driven by the demand of a data-driven scientific workflow running on dynamically provisioned 'slices' spanning multiple ExoGENI racks that are interconnected using dynamically provisioned connections from Internet2 and ESnet. We showed how a virtual Software Defined Exchange (SDX) platform, instantiated on ExoGENI, provides additional functionality for management of scientific workflows. We demonstrated how tools developed in the DoE Panorama project can enable the Pegasus Workflow Management System to monitor and manipulate network connectivity and performance. We used a representative, data-intensive genome science workflow as a driving use case to showcase the above capabilities.

PANORAMA: Predictive Modeling and Diagnostic Monitoring of Extreme Science Workflows

Scientific workflows are now being used in a number of scientific domains including astronomy, bioinformatics, climate modeling, earth science, civil engineering, physics, and many others. Unlike monolithic applications, workflows often run across heterogeneous resources distributed across wide area networks. Some workflow tasks may require high performance computing resources, while others can run efficiently on high throughput computing systems. Workflows also access data from potentially different data repositories and use data, often represented as files to communicate between the workflow components. As the result of the data access patterns, workflow performance can be greatly influenced by the performance of networks and storage devices.

Up to now workflow performance studies have focused on modeling and measuring the performance of individual tasks, primarily taking into account the behavior of computational tasks, often ignoring data management jobs. In turn much work in workflow scheduling and resource provisioning for workflow application has focused on the managing computations and to some degree ignoring data movement and storage. Today’s workflow monitoring tools again focus primarily on task monitoring, viewing data management tasks as black boxes. At the same time, network monitoring tools have focused on low-level network performance that cannot be easily correlated with the workflows utilizing the network.

The main questions this work aims to address are: 1) how to develop analytical models that can  predict the behavior of complex, data-aware scientific workflows executing in extreme scale infrastructures? 2) what monitoring information and information analysis is needed for performance prediction and anomaly detection in scientific workflow execution?, and 3) how to adapt the workflow execution and the infrastructure to achieve the potential performance predicted by the models. These questions will be addressed within the context of two DOE applications: Accelerated Climate Modeling for Energy (ACME), which processes large amount of community data and Spallation Neutron Source (SNS), which produces rich experimental data used in a variety of complex analysis.

The program of research focuses on data-aware workflow performance modeling, monitoring, and analysis and integrates this diverse information into knowledge about workflow behavior that can inform the scientist and the infrastructure providers about the observed performance issues and their causes.  This work will develop end-to-end workflow-level analytical models that capture the behavior of the workflow tasks performance on a variety of systems as well as workflow data movement and storage across different networks and devices. The analytical models will be coupled with simulation-based models to increase fidelity of the predictions in dynamic environments. The models will be validated through experiments on DOE infrastructures (such as the ESnet testbed and production infrastructure, the ORNL facilities), on distributed testbeds like ExoGENI, and through simulations.

The work will result in analytical models, workflow-level monitoring tools and monitoring recommendations for existing tools, which capture not only computational task behavior but also that of the data transfer and storage activities in the workflow. An analysis capability will correlate workflow monitoring information with resource performance measurements to provide a better understanding of which resources contributed to the observed behavior. The analytical models will be used to guide anomaly detection and diagnosis, resource management and adaptation, and infrastructure design and planning.

Evaluation: The project will develop workflows for the target applications and synthetic workflows to evaluate the accuracy and performance of the models and tools. The experiments will measure workflow performance at various level of detail and compare it to the model predictions. Failures and load will be introduced into the system and their effects on the accuracy of anomaly detection and diagnosis will be measured.

Project Partners

Ewa Deelman, University of Southern California
Brian Tierney, Lawrence Berkeley Laboratory
Jeffrey Vetter, Oak Ridge National Laboratory
Anirban Mandal, RENCI/UNC Chapel Hill
Christopher Carothers, Rensselaer Polytechnic Institute


The project aims at developing novel computer science techniques for workflow application modeling and management. The results of this work will have direct impact on DOE-relevant science.

The super cell of b NaLaF4 illustrates occupational structural disorder