076. How long does it take Part 2

The biggest takeaway here is that the runtime of GenotypeGVCFs scales down almost linearly with how widely you parallelize it, which is obviously great news if you’re in a rush and you have access to lots of machines. But pay attention here to the meaning of “thread count” in the context of the paper! As a reminder, most of the parallelization it presents is achieved through scatter-gather (parallelizing over predetermined genomic intervals), not by multithreading using `-nt` and/or `nct`. In our own production pipelines we don’t use `-nt`/`-nct` multithreading at all, and in GATK4 we’re abandoning them and replacing the functionality with Spark support wherever it makes sense. Why am I pointing this out here? Because we’re finding that GenotypeGVCFs is especially difficult to parallelize through multithreading, due to the complexity of dealing with overlapping events across multiple samples (the occurrence of which increases with cohort size). In the recent GATK 3.7 release, we added some functionality to deal better with overlapping deletions — and now we’re getting reports that this breaks when multithreading is turned on (cue the poop emoji). The safest way to deal with this? Don’t use multithreading with GenotypeGVCFs; use scatter-gather instead (ask me how in the comments).

Also, don’t parallelize VQSR. Look at the graph; it’s not worth it. VQSR needs to see all of the things most of the time.

Finally, I should add that having the exome numbers to compare to the WGS numbers is a big upgrade — it really gives you sense of scale of the practical implications of choosing to work with one datatype versus the other. All other sciencey considerations being equal (which they’re not, but let’s pretend) the computational resource commitment is massively different. Which is hardly news to our Ops team that processed Daniel MacArthur’s ludicrously large [gnomAD](http://gnomad.broadinstitute.org/) dataset, let me tell you — for reference, the final joint VCF on that was ~22TB for 20K genomes. That’s a big part of why we run our whole genomes on the cloud. It’s real Big Data, no hype needed.

Updated on 2017-12-21

From ying_sheng_1 on 2017-01-20

Nice presentation. Is it possible to have these data for GATK4?

From Geraldine_VdAuwera on 2017-01-22

We haven’t done this for GATK4 yet but we plan to do it once we have the full pipeline validated in GATK4. Stay tuned for an upcoming announcement of our calendar for GATK4 release.

From Gossie on 2017-11-28

Thanks for your presentation.

I have two questions:

1. How to parallelize GenotypeGVCFs by scatter-gather? (I guess you want us to ask ;) )

2. In White paper , it used the following arguments:

```

GenotypeGVCFs: Merges gVCF’s to create a genotyped VCF:

-nt “NUMTHREADS“

-R “GENOMEREF“

-D “DBSN VCF“

-V “input“

```

It used `-nt` which is different with your presentation, so I am confused.

Looking forward to your reply,

Thank you!

From Sheila on 2017-11-30

@Gossie

Hi,

I will ask Geraldine to get back to you soon.

-Sheila

From Gossie on 2017-12-01

Thank you @Sheila

From Sheila on 2017-12-06

@Gossie

Hi again,

Another teammate jumped in. It seems “For the Joint Analysis pipeline, VariantRecalibrator is currently unable to conduct process level parallelism and a comparison between both thread and process level parallelism techniques for the rest of the tools showed no significant improvement in time. Thus, all the tools in the Joint Analysis portion uses GATK’s integrated -nt argument to apply thread level parallelism.” This is from the [white paper](https://www.intel.com/content/www/us/en/healthcare-it/solutions/documents/deploying-gatk-best-practices-paper.html).

I hope that helps.

-Sheila

From Geraldine_VdAuwera on 2017-12-06

To clarify @Sheila’s comment, we realized that there was indeed a contradiction between my blog post and the white paper. It seems I misunderstood a communication I had at the time with the authors of the white paper. We had discussed the steps earlier in the pipeline that are parallelized using scatter-gather, and I thought that applied to all steps, but the joint calling part (including GenotypeGVCFs) is in fact parallelized using -nt. I will amend my blog post accordingly. My apologies for the confusion, and thank you for pointing out this error!

From Gossie on 2017-12-09

I see.

Thanks for your replies. Sheila Geraldine_VdAuwera