howto Upload files to your Google bucket

IMPORTANT: This is the legacy GATK documentation. This information is only valid until Dec 31st 2019. For latest documentation and forum click here

created by Tiffany_at_Broad

on 2017-10-31

In order to add data to your workspace, you need to upload your data files to the workspace bucket. You can upload data to your bucket through either the Google Cloud Platform Console, Google’s command line utility gsutil, or the FireCloud DataShuttle.

Watch the video demo or read more below.

Google Cloud Platform Console

1. To access your workspace's bucket, navigate to your Workspace Summary tab, the Storage and Analysis section, and click on the bucket URL.
2. Once you are in a bucket, you can click Upload Files or to download the file, click on its name.

gsutil

First, install gsutil to your local computer. The Google Cloud SDK installation includes gsutil. To install Google Cloud SDK

1. You can run the following command using bash shells in your Terminal: curl https://sdk.cloud.google.com | bash Or download google-cloud-sdk.zip or google-cloud-sdk.tar.gz and unpack it. Note: The command is only supported in bash shells.
2. Restart your shell: exec -l $SHELL or open a new bash shell in your Terminal.
3. Run gcloud init to authenticate, set up a default configuration, and clone the project's Git repository.

Before uploading data using gsutil, you can list buckets you have access to by running: * gsutil ls * gsutil ls -p [project name] to list buckets for a specific project

1. To upload data to a bucket run this command (you must be an Owner or Writer of the workspace to upload) :
2. Run gsutil cp [local file path] [bucket URL]

The bucket URL is the path to your file or folder in the Google Cloud SDK. It will look like gs://[bucket name], e.g. gs://jntest10052015 or for folders within a bucket, gs://[bucket name]/[folder name], e.g., gs://jntest10052015/gene_files.

Note: To download data from a bucket, run gsutil cp [bucket URL]/[file name], e.g., gs://jntest10052015/HCC1143.100gene250bp_pad.bam.

FireCloud DataShuttle

The FireCloud DataShuttle allows users to easily browse files, download and upload data directly between FireCloud workspaces & Google buckets and your local drives, and monitor the status of these transfers. It was developed to facilitate the work of researchers and project managers who transfer a high volume of files.

Download the FireCloud DataShuttle.

Updated on 2018-05-21