Quick Start Part 7 Workspace gt Monitor

IMPORTANT: This is the legacy GATK documentation. This information is only valid until Dec 31st 2019. For latest documentation and forum click here

created by Geraldine_VdAuwera

on 2018-01-07

Welcome to your workspace operations dashboard, where you can check the status of past and current pipelines run, drill down into various views of what’s going on and find direct links to all input and output files involved.

Your analyses are organized and reported on following this hierarchical structure:

• Submission: a collection of workflows that you have submitted in one batch.
• Workflow: in this context, the term workflow refers to a particular run of a method on a specific dataset.
• Task: the lowest level of analysis reporting representing individual calls/jobs made during workflow execution.

By default the Monitor page lists all submissions that have been made so far in a workspace along with their status; it is possible to filter the list, but it is not possible to delete any submissions. Similarly, it is not possible to delete workflows within a submission. For guidance relating to deleting files that belong to past submissions, please see the forum.

After you’ve launched a new submission, the Monitor page is loaded for you automatically so you can see the list of one or more workflows that are contained in the submission, along with their current status and links to further information. You can always go up one level to view all submissions by clicking on the Monitor tab – but for now let’s assume we just launched an analysis and we want to drill down into the status of a particular workflow.

Monitoring status

The workflows can be in the following states: Queued, Launching, Submitted, Running, Aborting, Succeeded, Failed or Aborted. When a workflow is in a Queued, Launching or Submitted state, the first three sequential statuses, the workflows are being handed off from FireCloud to Cromwell. More on this later…

Following Submitted is “Running;” it means that the commands specified in the method WDL script are being executed on virtual machines.

The Happy Path diagram below shows how each task status affects the workflow and analysis submission status -- in the best case scenario where all your tasks and workflows succeed. If you do not see a diagram below, please go here

Note: Aborted and Aborting status will display for analysis submissions, workflows, and tasks if you have requested a workflow to be aborted. These are not pictured here.

When all the tasks reach Done successfully, the workflow will be updated to Succeeded, and the Monitor page will show the submission as Done. You can look for outputs in the data model (via the Data tab) if you filled out the Outputs section (this.X) of the method configuration. Remember “this” refers to what data entity you are running your method on. If you chose participant and called an output this.participant_file, a new column will display in the participant section called participant_file and a link to the output will be put there, while the actual output file will be saved to the workspace’s bucket.

You can also view outputs on the Monitor page by clicking on the method configuration listed in the table, opening up the workflow you want to view, and selecting Outputs.

Behind the scenes

Under the hood, quite a lot is happening when you launch an analysis: various system components kick into gear to ensure that your submission of one or more workflows gets properly assembled and when that’s done, that individual task gets dispatched to the Google Compute Engine for execution. Meanwhile on the surface, FireCloud automatically takes you to the Monitor page where you can view the status of your workflow(s) and monitor how the work is progressing. If systems could talk, it would kind of look like this:

1. FireCloud takes the workflow specified in the WDL and asks Cromwell to run it.
2. Cromwell asks the Google Pipelines API (PAPI) to launch each task in the workflow when the inputs become available. Cromwell is responsible for managing the sequence of the tasks/jobs.
3. PAPI starts a virtual machine per task and provides the inputs; the WDL specifies what it should do, the environment to do it in (the Docker image), and requests the outputs when it is done. Each virtual machine’s (VM) requirements can be specified in the task (RAM, disk space, memory size, number of CPUs). Once the task is done, PAPI will shut down the VM.
4. The Docker required for each task will be pulled to the virtual machine along with any inputs from Google buckets. When the output is produced, it will be put in the Google bucket of the workspace where the analysis was launched. Links to the outputs will be written back to that workspace’s data model.

Troubleshooting

Sometimes your workflow fails (oh, the humanity!) and you need to troubleshoot. To learn about what is failing in your workspace, you can open the failed task to view more. Other than the Failure message, you can check out the task’s stdout log, stderr log, JES log (JES is now named PAPI), or Operation metadata to piece together what happened.

Debugging can be challenging when you're new to a system, so if you need some help, the following resources are available in the User Guide:

• Solutions to problems: A good place to look for troubleshooting tips and workarounds for commonly reported issues
• Bugs & Feature Requests: List of known bugs, limitations, and requested enhancements.
• Forum: Ask the support team a question directly or see if other users have asked your question. The support team tries to respond to every question within one business day.

You can access the User Guide or Forum while in FireCloud at any time by clicking the question mark in the top right corner.

Updated on 2018-04-24