created by jneff
on 2016-01-28
This article is in the process of being deprecated and replaced by a more cohesive guide for working with Docker images in FireCloud.
Please note that this post is intended for internal Broad users only.
Creating custom analyses to run within FireCloud requires both Workflow Description Language (WDL) and Docker images. FireCloud uses WDL to describe one or more tasks, including commands. WDL can refer to Docker images that package the applications needed to run your pipeline and methods into a discrete environment.
Rather than create a Docker image in a one-off manner and push it to Docker Hub, you should always create a Dockerfile. Dockerfiles describe the software environments and commands to run for the Docker image. Dockerfiles also allow version-controlled changes, the ability to see exactly what is on the image, and the ability to recreate the image as needed.
For a basic tutorial on creating Dockerfiles and Docker images, please go here. You can also check out Docker’s best practices for building Dockerfiles.
Below are some recommendations to help your group get started. If you respond to this topic, we can provide suggestions specific to your project.
1. Request a Broad github repo through Help@broadinstitute.org that will contain your group's WDLs and Dockerfiles. If you have both public and private WDLs/Dockerfiles, you may want to create two repos: one for public pipelines and another for private pipelines. If your goal is to publish a public pipeline, you may want to consider the implications of starting with a private repo (i.e., if tools you want to make public are embedded with other private tools).
2. In general, we encourage you to create as few Docker images as possible, and try to reuse images that serve most of your purposes. However, you may want to consider trade-offs if you are building a monolithic Docker image versus several smaller Docker images. For example, monolithic Docker images are typically easier to manage, but tend to have longer download times and may require artificial coupling between applications and software versions (i.e., upgrading one application forces upgrades in an unrelated application).
3. For each Docker image, create a Dockerfile and place it within a subdirectory in your github repo. You may choose to organize WDLs within these directories, or in other directories, but each Dockerfile must be within its own directory.
4. Request a Broad Docker Hub repo for each Dockerfile through Help@broadinstitute.org. Please choose a good name that is easily identifiable, as they will all be under the broadinstitute Docker Hub page. For instance teamnamedockername (e.g., cgamutsig).
Updated on 2017-12-14
From kcibul on 2016-02-07
Nice writeup — if there was an example GitHub repo we could share containing WDLs and Dockerfiles as an example, that might really help drive the point home
From jneff on 2016-02-08
Agreed. For now, here is a link to the Broad Institute’s Docker Hub page: [https://hub.docker.com/r/broadinstitute/](https://hub.docker.com/r/broadinstitute/).
As to WDL, users can click on Payload: expand to view the WDL for any public method in the Method Repository. There are also some WDL examples in this GitHub topic: [https://github.com/broadinstitute/wdl#getting-started-with-wdl](https://github.com/broadinstitute/wdl#getting-started-with-wdl), as well as in the (private) dsde-pipelines GitHub repo.
From dshih on 2016-06-19
If I understand docker correctly, users who do `docker run` need to be part of the `docker` group (or another group specified by `-G` when the docker daemon was started by an admin), and this `docker` group essentially has root privilege:
http://www.projectatomic.io/blog/2015/08/why-we-dont-let-non-root-users-run-docker-in-centos-fedora-or-rhel
http://reventlov.com/advisories/using-the-docker-command-to-root-the-host
I don’t suppose it would be possible to run `docker` on a Broad VM or cluster?
So is the idea to use `docker` locally on your own machine for testing and deploy it to Google JES for production?
From jneff on 2016-06-21
It sounds like you had a chance to discuss with Chet. If you have any further questions, please let us know.
You can also check out files from our recent (https://us.v-cdn.net/5019796/uploads/FileUpload/45/2a47a9e1e189205d6eee635261cc29.zip). In this folder you will find tutorials that cover the basic steps for FireCloud tool development.