Transferring Files

Data transfer servers

Case has configured an alternate route for research data to be transferred with off-campus servers. This is known as our Science DMZ, and data routed here is not subject to packet-level inspection. At this time, only certain servers located within the KSL data center have access to the Science DMZ. Use dtn[2-3].case.edu to use Globus, iCommands or other tools to transfer data via the Science DMZ to external servers. Use hpctransfer.case.edu to move data between your local system and the cluster data storage. The server dtn[2-3].case.edu only supports graphical access through Globus.

Important Notes

• While transferring huge files, makes sure that you are in your quota limit. Refer to HPC Guide to Storage Quota.

• File transfer command like sftp works fine internally in HPC using Case Network or connecting via VPN, however, for TOPMED like application that access the data externally using sftp, use HPC web interface https://ondemand.case.edu (HPC Guide to OnDemand) that doesn't require VPN access.

Transferring Files

There are several methods to transfer data with the cluster data storage. 

• Globus Online (GO) using dtn[2-3] node (Recommended for larger files and faster transfer) - click here for setting-up GO

  • iCommands by iRods; using dtn[2-3] node (Recommended for larger files and faster transfer)

• For your Windows machines, here are some of the options (you can download):

  • WinSCP - Opens Source GUI Secure File Transfer Protocol (sFTP) Clinet

  • MobaXterm - It has visual enabled SSH, SFTP, RDP sessions under a single hood to enable easy file transfer and remote desktop

  • FileZilla - Open Source GNU/GPL FTP Clinet

  • PSFTP - Putty's secured FTP

  • Window with Cygwin - OpenSSH

• For your Mac machines, here are some applications you can use:

  • Cyberduck

  • FileZilla

• For Linux box with OpenSSH: Cyberduck, other open source scp-compatible tools

• To back up the files on the HPC to Google drive, please use Rclone - Google Drive 

• WebDrive

Command-line Transfer

scp command

usage: scp -r <source> <destination>

At the Command Line prompt of your desktop or laptop computer, type the command (e.g. in MobaXterm the ~ (home) is /home/mobaxterm which lists local directories such as  Desktop , LauncherFolder,  MyDocuments etc.)

scp -r <CaseID>@<login-node>:/home/<Caseid>/<source-files> <destination> # Copy from HPC to your local PC

scp -r <file or directory-to-be-copied> <CaseID>@<login-node>:/home/<user-id>/<location-for-file-to-transfer> # From your PC to HPC

Graphical Interface Remote Copy Application

Globus Online

GO Enables faster transfer. Click HERE for setting-up GO.

iCommands

A parallel file transfer tool developed for transfers with the CyVerse Data Store. Setup and usage references available here.

WebDrive

WebDrive can help transfer files (SFTP) mapping drive letters to webservers (e.g. hpctransfer.case.edu) on top of providing cloud access (Amazon S3), Google Drive, Dropbox, and more.

• Download and Install the WebDrive Software from Software Center.

• Choose SFTP as the type of server you want to connect to

• In the URL/Address, type the name of the SFTP server (e.g. hpctransfer.case.edu)

• Type your Case (SSO) username and Password.

• Choose the drive letter for that server

Utilizing Cloud Storage with the HPC

Cloud storage access from HPC/RCCI storage is achieved using rclone (rclone.org). Rclone is an implementation of 'rsync' using the API tools provided by a large group of the cloud storage vendors. The API uses "web protocols" for transfer, and is not really similar to scp as described above.

Object storage is the basis for the large cloud storage services. This is distinct from the block storage model implemented in HPC parallel file systems. With object storage, the file (along with metadata) is treated as a single object, and is manipulated in its entirety on storage. With block storage, files are written onto preformatted blocks on disk, and can be addressed and modified at the block level.

The combination of these two factors (web data transfer protocols and asymetric storage devices) leads to modest data transfer rates, compared with scp or gridftp (e.g. Globus) between source and destination block storage systems.

RClone - Google Drive

Rclone is a very handy tool to access your folders in a google drive. Click here for details

Using Box at HPC

Rclone can also be used to access Box from the HPC. 

Mounting Research Storage to remote computers

Mac computers

The Mac OS provides native support to access the Research Storage as a remote drive.

Basic instructions are:

1.  Click on Finder

2.  Go -> Connect to Server..

3.  Paste: "smb://ads.case.edu/rc" into the text box -> Click Connect

4.  Enter <userid> and password.

More detailed instructions:  https://docs.google.com/document/d/1MWYtZEXXtESL8b88ajb9qHo8kG-iIj6X-775Lrbqblc/edit

Windows

Windows OS also provides support for drive mapping. Please review the following document for detailed instructions:

https://docs.google.com/document/d/1VVT9jJrxuF7f5AWC98thGhYcjLbwoItwOb3Zg9DFkDE/edit