Original post: https://rcc.fsu.edu/docs/globus

Using Globus to Share Data

We provide access to a service called Globus to transfer data to and from the HPC Storage (Panasas and Lustre). Globus can be used in a number of different ways:

• • Reliably transfer a large amount of data from Panasas or Lustre to another machine or server.

  • Reliably transfer a large amount of data onto Panasas or Lustre.

  • Allow a colleague with an RCC Account to copy data from Panasas or Lustre

  • Setup automated replication of data between Panasas/Lustre and another server or computer.

As an alternative to SFTP, Globus is a more fully-featured and reliable tool for transferring large datasets.

Setting up a Globus Account and Endpoint

Before you can transfer data to and from RCC systems, you must setup a Globus Account and configure the target computer (personal computer or server) as a Globus Endpoint.

• 1. Visit http://globus.org and sign up for an account. Then sign in.

  2. Browse to the Manage Endpoints screen.

  3. Click the Add Globus Connect Personal link.

  4. When prompted, create an Endpoint name, and click Generate Setup Key

  5. Take note of the Setup Key when it appears.

  6. Click the appropriate client download button for your operating system.

  7. Install the client and follow the on-screen instructions. You'll need your setup key for this step.

Transferring Data

Once you have setup an account and endpoint, you can transfer data anytime.

• 1. With the Globus Client running, browse to the Tranfser Files screen at http://globus.org.

  2. On the right-hand side of the screen, enter the name of your endpoint (or click the "..." button to find your endpoint in a list)

  3. Select the path in your file tree that you wish to transfer your files to or from.

  4. On the left-hand site, enter either fsurcc#panfs or fsurcc#lustre, depending on which system your data is in.

  5. In the authorization dialog box that appears, enter your RCC username and password. You do not need to alter any other fields in this dialog box.

  6. You will see the root of the filesystem appear. If you wish to see your home directory, place a tilde (~) in the Pathbox, and press Go.

  7. Select the files and folders you wish to transfer. Then, click the large blue arrow in the direction that you wish to transfer files.

  8. Your files will now be queued for transfer. You can monitor the progress of your transfer by viewing the View Transfer Activity Page on the Globus website.

For more information about using Globus, refer to the official documentation.

Using the Globus CLI

It is possible to initiate transfers and manage endpoints using the Globus Command Line Interface via SSH. This is ideal for transferring files to and from servers with no GUI, and for automating the transfer of files with scripts. All Globus CLI commands take the following form:

$ ssh <username>@cli.globusonline.org <command> <options> <params>

Prerequisites

Before you can use the Globus CLI, you must enable it.

• 1. Sign in to your Globus account at http://globus.org

  2. Navigate to the Manage Identities page

  3. Click "add linked identity" and then "Add SSH Public Key"

If you do not already have an SSH public key, follow the directions on the Globus CLI FAQ to create one.

Transferring Files

The Globus CLI offers two commands for transferring files: transfer, and scp. scp follows a syntax similar to that of the unix utility scp, but transfer is more sophisticated.

To transfer a file called myfile.txt from in the directory /my/lustre/dir/ on endpoint fsurcc#lustre to the directory /my/panfs/dir/ on endpoint fsurcc#panfs using the scp command and the username cberkley would look like this:

$ ssh cberkley@cli.globusonline.org scp fsurcc#lustre:/my/lustre/dir/myfile.txt fsurcc#panfs:/my/panfs/dir/myfile.txt

By default an scp transfer will be canceled if your SSH session is disconnected. Using the -D option will detach the scp task, which will now run in the background even if the session is disconnected. The -r option may be used to copy files recursively.

Getting Help

More information about the Globus CLI may be found through the Globus Using the CLI guide, or through the Globus CLI itself

$ ssh <username>@cli.globusonline.org help