File transfer from/to the cluster

For global issues in file transfer contact:helpdesk@ggus.org 

Transfer files to the cluster

These instructions are for transferring small quantities of personal files to or from your working area on the cluster using scp or sftp. If you wish to transfer files from FNAL's or CERN's storage element, please use xrootd. If you wish to transfer data from your space in /data, please break transfers up into small chunks, not exceeding 100MB in size. Files which are stored in the storage element (/store, /mnt/hadoop) can be xrootd.

Forbidden Hadoop Operations:

Below are frequently found operations from users that should never be happening under

/mnt/hadoop (soft linked as  /store):

    1) cd into any subdirectory

    2) rsync of anything under /mnt/hadoop

    3) putting libraries or executable anywhere under /mnt/hadoop

    4) writing lots of small log or text files

    5) writing more that ~4K files into single directory

    6) rm of anything (you need to use srm-rm tools with your grid certificate, or contact sysadmins with a list of files to remove)

    Please make sure that you do not try to do anything from the list above

Copy To/From a Linux machine:

To the cluster:

scp filename.ext username@hepcms.umd.edu:~/filename.ext

OR

scp filename.ext username@hepcms.umd.edu:/data/users/username/filename.ext

From the cluster:

scp username@hepcms.umd.edu:~/filename.ext filename.ext

OR

scp username@hepcms.umd.edu:/data/users/username/filename.ext filename.ext

scp accepts wildcards (*) and -r for recursive functionality.

To/From a Windows machine:

FileZilla is a useful Windows application for transferring files using various protocols. Once installed, select File->Site Manager. Click on the "New Site" button and enter hepcms.umd.edu as the host. Select SFTP using SSH2 as the Servertype. Select Logontype Normal and enter your user name and password if desired. Click the "Connect" button. Transfer files by dragging them from the left directory to the right or vice-versa.

WinSCP is another convenient file copy utility, version 4.0.7 is definitely compatible with the Kerberos-enabled PuTTY client linked from the FNAL website, though newer releases have also been confirmed to work (4.2.9 confirmed OK).

 Transfer data using XROOTD

 voms-proxy-init -voms cms

for a dataset 

/store/group/phys_exotica/EmergingJets/Analysis-20161012 v0/ModelA/EmergingJets_ModelA_TuneCUETP8M1_13TeV_pythia8Mod/Analysis-20161012/161018_065904/ntuple_merged_ModelA.root 

Transfer to hepcms hadoop

xrdcp root://cmsxrootd.fnal.gov//store/group/phys_exotica/EmergingJets/Analysis-20161012-v0/ModelA/EmergingJets_ModelA_TuneCUETP8M1_13TeV_pythia8Mod/Analysis-20161012/161018_065904/ntuple_merged_ModelA.root  /mnt/hadoop/cms/store/ntuple_merged_ModelA.root

tranfer to hepcms /data/user area do

xrdcp root://cmsxrootd.fnal.gov//store/group/phys_exotica/EmergingJets/Analysis-20161012-v0/ModelA/EmergingJets_ModelA_TuneCUETP8M1_13TeV_pythia8Mod/Analysis-20161012/161018_065904/ntuple_merged_ModelA.root /data/users/jabeen/ntuple_merged_ModelA.root

List files 

 xrdfs root://hepcms-se.umd.edu:1094/ ls /store/user/bhatti/

Make directory

 xrdfs root://hepcms-se.umd.edu:1094/ mkdir /store/user/bhatti/test

Remove directory:

 xrdfs root://hepcms-se.umd.edu:1094/ rmdir /store/user/bhatti/test

Transfer data via the lcg-cp protocol  [obsolete, use xrootd]

Generally, to transfer datasets registered in the global DBS, use FileMover to transfer individual files registered in the global DBS, or use scp/sftp to transfer individual files from UMD (/data). SRM and lcg-cp work to manage a few files located in our SE (/store, /mnt/hadoop), or is a good option if you wish to transfer a few files inside someone's personal storage area on another site's storage element. We use the lcg-cp version of srm commands as they work after the OSG3 upgrade (May 2012). You will need to know the details for the storage element hosting the dataset; examples are provided for FNAL, CERN, and T3_US_UMD. DBS-registered data hosted at our site can be copied using exactly the filename path given in DBS (always starting with /store).

You will need your grid certificate and proxy to execute SRM commands.

• General srm-copy syntax

• Examples

• T3_US_UMD hadoop examples

General srm-copy syntax:

Warning: srm-copy command are not currently working after the OSG3 upgrade (May 2012) and this section will be replaced with equivalent lcg-cp commands once they are tested. A working command example (replace the bold with your file location):

lcg-cp -v --srm-timeout 3600 -b -D srmv2 --sendreceive-timeout 2400 --connect-timeout 300 file:///proc/cpuinfo srm://hepcms-0.umd.edu:8443/srm/v2/server'?'SFN=/mnt/hadoop/cms/store/user/username/testfile_May2012.txt

srm-copy functions as a typical copy command, where you specify the source, then the destination. srm-copy does not accept wildcards or recursive functionality.

If you are located at the UMD cluster (/home, /data) and wish to transfer data from a different site's storage element (SE):

srm-copy "srm://se-where-data-is-located:8443/se-path?SFN=/full-path/filename.ext" file:////full-path/filename.ext"

If you are located at a remote site, you may need to set up your environment to get the srm-copy binary in your PATH (which srm-copy to verify). To get a file hosted at the UMD SE:

srm-copy "srm://hepcms-0.umd.edu:8443/srm/v2/server?SFN=/path/filename.ext" file:////full-path/filename.ext

A few notes:

• • scp and sftp are much easier ways to get data to and from the cluster in the /data area, as our storage element is a normal disk system and does not require srm-copy commands to retrieve data.

  • Due to grid authentication, you need to use srm-copy, srm-rm, and lcg-cp commands to manipulate files in your local storage element area (/store, /mnt/hadoop).

  • Avoid using any of these commands (srm-copy, scp, or sftp) for large transfers (10GB+) and contact the System administrators and/or Nick Hadley if you must do so.

  • Call srm-copy --help to view further options.

  • Transfers using the srm protocol to or from afs areas will not work. Work in a different area or move the file using conventional commands to a different area first. The UMD cluster does not have any afs mounts.

  • CERN no longer has lcg commands installed on lxplus, however, the commands below are kept for reference if useful as they should work at Maryland.

    • CERN's Castor did not used to support srmcp or srm-copy. It may not, you may need to use gfal-copy.

    • To use lcg-cp to get files to and from Castor. lcg-cp syntax for CERN takes the form:

  • lcg-cp --verbose -b -D srmv2 source destination

    • where CERN's CMS Castor service runs at:

  • srm://srm-cms.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/...

Examples:

Examples for CERN & FNAL are provided below. Note that the username at one site may not be the same as at the other site.

Transferring a file from FNAL's dCache user area (/pnfs/cms/WAX/11/store/user/username/file.ext) to your UMD home area (/home/username/file.ext):

voms-proxy-init -voms cms

cd ~

voms-proxy-init -voms cms

lcg-cp --verbose -b -D srmv2 "srm://srm-cms.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/cms/store/user/username/file.ext" file://`pwd`/file.ext

Transferring a file from one of CERN's Castor user areas (/castor/cern.ch/cms/store/caf/user/username/file.ext) to /tmp, which is garbage collected for week old files and is not network mounted, so only accessible from the node you are working from (/tmp/file.ext) (Warning: do not fill up /tmp as usage on that interactive node will fail):

voms-proxy-init -voms cms

lcg-cp --verbose -b -D srmv2 "srm://srm-cms.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/cms/store/caf/user/username/file.ext" file:///tmp/file.ext

Transferring a file from another of CERN's Castor user areas (/castor/cern.ch/user/u/username/file.ext) to your UMD /data area (/data/users/username/file.ext):

voms-proxy-init -voms cms

lcg-cp --verbose -b -D srmv2 "srm://srm-cms.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/user/u/username/file.ext"file:///data/users/username/file.ext

Transferring a file from another of CERN's Castor user areas (/castor/cern.ch/user/u/username/file.ext) to your UMD /mnt/hadoop storage element area (/mnt/hadoop/cms/store/user/username/file.ext):

voms-proxy-init -voms cms

lcg-cp --verbose -b -D srmv2 "srm://srm-cms.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/user/u/username/file.ext""srm://hepcms-0.umd.edu:8443/srm/managerv2?SFN=/mnt/hadoop/cms/store/user/username/file.ext"

Transfer file from CERN EOS area to Maryland:

lcg-cp -v --srm-timeout 36000000 -b -D srmv2 --sendreceive-timeout 24000000 --connect-timeout 30000000 srm://srm-eoscms.cern.ch:8443/srm/v2/server'?'SFN=//eos/cms/store/group/phys_heavyions/username/filename.root srm://hepcms-0.umd.edu:8443/srm/v2/server'?'SFN=/mnt/hadoop/cms/store/user/username/filename.root

T3_US_UMD hadoop examples:   [SOON to be obsolete, use xrootd instead]

Examples for moving files within T3_US_UMD storage element (/hadoop, /store). 

Copying a file from your /data area (/data/users/username/) to your /mnt/hadoop storage element (/mnt/hadoop/cms/store/user/username/):

 eval `scram unsetenv -sh`; 

voms-proxy-init -voms cms

gfal-copy -p -v -t 180  filename.txt   'gsiftp://hepcms-gridftp2.umd.edu:2811/mnt/hadoop/cms/store/user/username/directory/'