Moving Data Between Local and Remote Machines

To move data and files between servers, it is recommended to use the rsync command, which is "a fast, versatile, remote (and local) file-copying tool." It is recommended to tar a directory before moving it.

What is 'tar?'

The tar command is used to compress and archive files and directories to be smaller and easier to file transfer, and this result is frequently called a tarball.

Creating a tarball

Say there is a directory called something/ in the working directory. This will create (-c) a tarball (.tar) and then compress (-z) it using gzip (.gz) of the directory something/, and name it (-f) something.tar.gz, while taking in the directory to tar something/. It is HIGHLY recommended to tar an entire directory that is not the working directory.

$ ls

something/

$ tar -czf something.tar.gz ./something

$ ls

something/ something.tar.gz

Extracting a tarball

This will extract (-x) and undo gzip (-z) from tarball (-f) something.tar.gz. Once this is completed, the original tarball will remain, and the contents of the tarball, something/, will be in the working directory.

$ ls

something.tar.gz

$ tar -xzf something.tar.gz

$ ls

something/ something.tar.gz

File Transfer

rsync uses the syntax rsync <source> <destination> to copy files. rsync will make the file copy verbose (-v), and make any data output more human readable (-h).

Moving files from a local machine to a remote RC machine

This will copy the tarball something.tar.gz to Agamede, into the user's home directory.

rsync -vh ./something.tar.gz <odinID>@agamede.rc.pdx.edu:

Moving files from a remote RC machine to another remote RC machine

This will move something.tar.gz on agamede to /scratch on Coeus through a login node.

$ ssh <odinID>@agamede.rc.pdx.edu

$ pwd

/home/<odinID>/

$ ls

something.tar.gz

$ rsync -vh ./something.tar.gz <odinID>@login1.coeus.rc.pdx.edu:/scratch/

Moving files from a remote RC machine to a local machine

The below is ran on a local machine.

$ rsync -vh <odinID>@login1.coeus.rc.pdx:/scratch/something.tar.gz .

$ ls

something.tar.gz