File transfer from/to the cluster

For global issues in file transfer contact:helpdesk@ggus.org 

Transfer files to the cluster

These instructions are for transferring small quantities of personal files to or from your working area on the cluster using scp or sftp. If you wish to transfer files from FNAL's or CERN's storage element, please use xrootd. If you wish to transfer data from your space in /data, please break transfers up into small chunks, not exceeding 100MB in size. Files which are stored in the storage element (/store, /mnt/hadoop) can be xrootd.

Forbidden Hadoop Operations:

Below are frequently found operations from users that should never be happening under

/mnt/hadoop (soft linked as  /store):

    1) cd into any subdirectory

    2) rsync of anything under /mnt/hadoop

    3) putting libraries or executable anywhere under /mnt/hadoop

    4) writing lots of small log or text files

    5) writing more that ~4K files into single directory

    6) rm of anything (you need to use srm-rm tools with your grid certificate, or contact sysadmins with a list of files to remove)

    Please make sure that you do not try to do anything from the list above

Copy To/From a Linux machine:

To the cluster:

scp filename.ext username@hepcms.umd.edu:~/filename.ext

OR

scp filename.ext username@hepcms.umd.edu:/data/users/username/filename.ext

From the cluster:

scp username@hepcms.umd.edu:~/filename.ext filename.ext

OR

scp username@hepcms.umd.edu:/data/users/username/filename.ext filename.ext

scp accepts wildcards (*) and -r for recursive functionality.

To/From a Windows machine:

FileZilla is a useful Windows application for transferring files using various protocols. Once installed, select File->Site Manager. Click on the "New Site" button and enter hepcms.umd.edu as the host. Select SFTP using SSH2 as the Servertype. Select Logontype Normal and enter your user name and password if desired. Click the "Connect" button. Transfer files by dragging them from the left directory to the right or vice-versa.

WinSCP is another convenient file copy utility, version 4.0.7 is definitely compatible with the Kerberos-enabled PuTTY client linked from the FNAL website, though newer releases have also been confirmed to work (4.2.9 confirmed OK).

 Transfer data using XROOTD

 voms-proxy-init -voms cms

for a dataset 

/store/group/phys_exotica/EmergingJets/Analysis-20161012 v0/ModelA/EmergingJets_ModelA_TuneCUETP8M1_13TeV_pythia8Mod/Analysis-20161012/161018_065904/ntuple_merged_ModelA.root 

Transfer to hepcms hadoop

xrdcp root://cmsxrootd.fnal.gov//store/group/phys_exotica/EmergingJets/Analysis-20161012-v0/ModelA/EmergingJets_ModelA_TuneCUETP8M1_13TeV_pythia8Mod/Analysis-20161012/161018_065904/ntuple_merged_ModelA.root  /mnt/hadoop/cms/store/ntuple_merged_ModelA.root

tranfer to hepcms /data/user area do

xrdcp root://cmsxrootd.fnal.gov//store/group/phys_exotica/EmergingJets/Analysis-20161012-v0/ModelA/EmergingJets_ModelA_TuneCUETP8M1_13TeV_pythia8Mod/Analysis-20161012/161018_065904/ntuple_merged_ModelA.root /data/users/jabeen/ntuple_merged_ModelA.root

List files 

 xrdfs root://hepcms-se.umd.edu:1094/ ls /store/user/bhatti/

Make directory

 xrdfs root://hepcms-se.umd.edu:1094/ mkdir /store/user/bhatti/test

Remove directory:

 xrdfs root://hepcms-se.umd.edu:1094/ rmdir /store/user/bhatti/test