howto Publish a docker container image to Google Container Registry GCR

IMPORTANT: This is the legacy GATK documentation. This information is only valid until Dec 31st 2019. For latest documentation and forum click here

created by KateVoss

on 2018-03-05

This document explains how to create and utilize a public or private Google Container Registry (GCR) docker image from inside FireCloud. This guide is useful for migrating or mirroring existing images from other repositories such as DockerHub. For a primer on what Docker containers are for and related terminology, see this Dictionary entry.

Contents

1. Prerequisites
2. Create or obtain a container image
3. Tag image with registry name
4. Push your image to GCR (private)
5. Optional: make your image public
6. Share your image with a FireCloud group
7. Use the image in a workflow

Some of these instructions were adapted from Google GCR Documentation.

1. Prerequisites

• Install Docker (see this tutorial for guidance)
• Install Google Cloud SDK
• Google Billing account and Project
  • You need a Google Billing account to set up a new Google Project to store your images. This project must be different than your FireCloud Billing Project.
  • Once you have a Google Billing account, follow the "Before you begin" section of the GCR quickstart tutorial to create a Google project and a Google container repository.

2. Create or obtain a container image

To create a new image, follow the first section of this tutorial to create a docker image and tag an image.

If you already have an image you want to use and you have a local copy, simply continue to the next step (2. Tag image). If your image is stored in DockerHub and you want to migrate it to GCR, first you need to pull it locally with this command:

docker pull <REPOSITORY-NAME>/<IMAGE>:<TAG>

where <REPOSITORY-NAME> is the name of the repository where the docker is stored, <IMAGE> is the name of the image you are pulling, and <TAG> is a keyword or version number that you want to attach to identify a specific image.

Note that the default tag is latest. While this can be convenient for some purposes, we don't recommend using the "latest" tag when pipelining commands in real work, unless you're sure that you're willing to take the chance that the version of what you're running might change without warning. It's better practice to use explicit tags. both for pulling and pushing images.

3. Tag image with registry name

Before you push your Docker image to GCR, you need to tag it with its registry name. Tagging your Docker image with a registry name configures the docker push command to push the image to a specific location. The registry name format is: gcr.io/[PROJECT-ID]/[IMAGE] where [PROJECT-ID] is your Google Cloud Platform Console project ID and [IMAGE] is your image's name.

To tag your Docker image for Container Registry, run this command:

docker tag [IMAGE] gcr.io/[PROJECT-ID]/[IMAGE]

For example:

docker tag quickstart-image gcr.io/my-project/quickstart-image

You are now ready to push your image to GCR!

4. Push your image to GCR (private)

To push your Docker image to Container Registry, run this command:

docker push gcr.io/[PROJECT-ID]/[IMAGE]

For example:

docker push gcr.io/my-project/quickstart-image

You can view your freshly-pushed image in GCR by visiting the image’s registry at http://gcr.io/<PROJECT-ID>/<IMAGE>. You can also view your image where it is stored in the Google Bucket. The directory will be named artifacts.<PROJECT-ID>.appspot.com for images pushed to gcr.io/<PROJECT-ID>.

By default, Google stores your images privately so only authorized users have permission to use them. You have the option of sharing your images publicly as well; see the next step, which is optional.

5. Optional: make your image public

You can also make your images public, which has the benefit of allowing you to use them in FireCloud or directly through Cromwell without following any more steps in this tutorial.

To do this through the Google Console, go to the Google Bucket that you want to make public.

1. Check the box next to the name of the Bucket.
2. Click “SHOW INFO PANEL”.
3. Type allUsers in the “Add members” field.
4. In the “Select a role” drop-down menu select “Storage Object Viewer”, and click “Add”.

This will give pull (read-only) permissions to all users. You can also change the Bucket permissions with the Google Cloud SDK. Note that each Google Bucket is either entirely public or not; it is not possible to publicly serve only specific images. If you have specific images you want to make public while keeping the rest private, create a separate Bucket and GCR to make it public.

If in the future you change an image from public to private, be aware that any pipelines that use this formerly-public image will break for anyone who does not have access to the private image.

6. Share your image with a FireCloud group

You can share your images with a new group or an existing group within FireCloud. We strongly recommend that you share with a group (as opposed to an individual) if you ever plan on sharing your images in the future, as you can easily add new people to the FireCloud group. As an added benefit, they will have access to all of the images that you have ever shared with that group.

To do this, go to FireCloud and view your groups. To create a new group, click “Create New Group…” and follow the instructions to create and save a group.

Then you can simply give that group access to your GCR bucket as follows:

1. Copy the email address under “Email for Sharing Workspaces”, for example, GROUP_<YOUR-GROUP-NAME>@firecloud.org.
2. Go to the Google bucket and check the box next to the name of the Bucket containing the private images you want to use in FireCloud.
3. Click the “SHOW INFO PANEL”.
4. Paste the FireCloud group email address in the “Add members” field.
5. In the “Select a role” drop-down menu select “Storage Object Viewer”, and click “Add”.

This will give pull (read-only) permissions to FireCloud users who are in the FireCloud group.

7. Use the image in a workflow

The path to your image in GCR is gcr.io/<PROJECT-ID>/<IMAGE>, for example, gcr.io/my-project/quickstart-image; use this image path in your WDL workflow.

Additional Resources

• Overview of Google Container Registry
• Guide to Configuring Access Control for GCR

Updated on 2018-05-21