Submitting analysis jobs

Submit jobs to the cluster using Condor

The best way to take advantage of all the cluster CPU is to run jobs using Condor. Long-running interactive jobs that consume memory and CPU on the interactive nodes are highly discouraged and may be killed. Additionally, condor jobs should run for less than 24 hours, especially if users submit more than 50 jobs. Please split your jobs into smaller jobs. The current configuration of the cluster cannot handle multi-core jobs, instead it is 1 CPU per condor job.

Jobs are submitted via condor using the command:

condor_submit jdl_file

Details on other condor commands and possible values in the jdl file can be found in this Condor guide.

Here is a very simple example jdl file which run the script  test.sh (the script you would like to run) which is in area   /home/$USER}/condor_test/.

Save the lines below in  a file,  name the condor_test.jdl 

  # Here is a very simple submission scrip

  Universe = vanilla

   Executable = /home/$USER}/condor_test/test.sh 

  Output =/home/${USER}/condor_test/log/output_$(cluster)_$(process).stdout

  Error  =/home/${USER}/condor_test/log/output_$(cluster)_$(process).stderr

  Log    =/home/${USER}/condor_test/log/output_$(cluster)_$(process).log

  Should_Transfer_Files = YES

  When_to_transfer_output = ON_EXIT

  Queue    

  # end of script

Place this file in one of your directories on the cluster. Edit it for your desired output directory, then submit it to Condor:

condor_submit condor_test.jdl

You can watch the status of the jobs:

condor_q

If you are in interactive node 1 (in1) and you submitted your job in hepcms-rubin.umd.edu, you can do:

condor_q -name hepcms-rubin.umd.edu

Automatically re-run hanging condor jobs with built-in timer

Edit expected amount of time a script is supposed to run. If it is running for too long it might mean the script is hanging.

  1 # Here is a condor submission script

  2 # To automate job removal in case of a hang

  3 # retry if run time exceeds 2 hours, restart no more than 5 times

  4 # we should all use these in our job description files

  5 universe = vanilla

  6 Executable = /bin/sleep

  7 getenv = True

  8 MINUTE      = 60

  9 periodic_hold = (CurrentTime - JobCurrentStartDate) >= 120 * $(MINUTE)

 10 #optional: for rerunning

 11 #periodic_remove = (CurrentTime - JobCurrentStartDate) >= 500 * $(MINUTE)

 12 periodic_release = NumJobStarts<5

 13 Arguments = 100

 14 Queue

Automatic re-run failed jobs

try on_exit_remove expression or on_exit_hold

Example:

on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)

on_exit_hold = (ExitBySignal == False) && (ExitCode == 0)

periodic_release = NumJobStarts<5 && (CurrentTime - JobCurrentStartDate) >= 30 * $(MINUTE)

If you use  exit $exitcode to propagate exit code in shell script, you should use:

on_exit_hold = (ExitCode == 0)

Refer to HTCondor manual for more details on on_exit_remove and on_exit_hold: 

https://htcondor.readthedocs.io/en/stable/man-pages/condor_submit.html

Pace job start time

Pace job start time by 1 second each. If starting all jobs at the same moment causes problem this may be helpful. However, beware that higher delays might take long time for all jobs to start running (e.g. 300jobs * 60 sec = 5 hours), and if your code causes a race condition, it is a design problem and this configuration will not help.

next_job_start_delay=1

Condor Machine Requirement

It is important to specify job requirements so that they make use of the slots that can run them properly. This may be due to lack of mounted disk, OS and software environment, or simply jobs failling to run on specific node. To rule out certain node, use:

Rank = TARGET.Machine =!= "siab-1.umd.edu"

Condor Machine Ranking

Default ranking algorithm is given by:

NEGOTIATOR_PRE_JOB_RANK = (JobOverridesNegotiatorRank =!= True) * \

                          isUndefined(RemoteOwner) * (KFlops*(-SlotID) +\

                regexp("r720.*",machine)*5.0e6 + regexp("r510.*",machine)*1.0e6)

This means jobs will generally be equally distributed to all nodes, but with slight preference to r720 and r510s

This can by overriden by Rank expression in Job description jdl files. For example, to prefer one compute node:

+JobOverridesNegotiatorRank = True

Rank = TARGET.Machine =?= "compute-0-10.privnet"

With regular expressions we can select a group of nodes:

+JobOverridesNegotiatorRank = True

Rank = regexp("compute.*",TARGET.Machine)

Long Running Jobs

Jobs running longer than a day needs to add long running tag or they risk being kicked off. This is to prevent jobs with loop condition to run forever

Multi-thread Jobs

These are jobs that automatically scales with number of available threads in a machine. If they are submitted to normal slots they crowd out resources for other jobs

Email notification for condor jobs

Notify yourself through email. Options are always (all state changes), or complete (only send email when job completes)

  1 universe = vanilla

  2 Executable = /bin/sleep

  3 notify_user = username@umd.edu

  4 notification = always

  5 Arguments = 100

  6 Queue  

More Condor job description recipes

Please refer to the official HTCondor manual

http://research.cs.wisc.edu/htcondor/manual/v7.6/condor_submit.html

A Short Introduction to HTCondor

http://personalpages.to.infn.it/~gariazzo/htcondor/concepts.html

Condor Log Analyzer

Condor log analyzer is a handy way to check batch performance after the jobs finish. This is particularly useful if a lot of jobs are submitted

http://condorlog.cse.nd.edu/

CMSSW (cmsRun) with Condor

The following three files together are an example of how to submit CMSSW jobs to the cluster with Condor:

• condor-executable.sh

• condor-jobs.jdl

• condor-simple.py

To run these files, setup a CMSSW release and place condor-simple.py inside the src subdirectory. condor-simple.py has been verified to work in CMSSW_3_9_9, though it should work in most modern releases. To run other software (i.e., Madgraph), you do not need a CMSSW release area. condor-executable.sh and condor-jobs.jdl can be placed together in any directory. Edit condor-jobs.jdl with your desired output directory, a unique prefix name to help you identify the files, and your CMSSW release area. Make condor-executable.sh executable:

chmod +x condor-executable.sh

Then submit the jobs:

condor_submit condor-jobs.jdl

You can monitor the status of your jobs, they should take about 30 minutes to complete once they start running:

condor_q -submitter your_username

You can also watch condor_q:

watch -n 60 "condor_q -submitter your_username"

Once jobs are complete, the *.condor files will indicate if condor itself ran successfully, as well as providing the exit code of the CMSSW job. The *.stdout and *.stderr files will show output from condor-executable.sh and the *.log files will show output from CMSSW itself.

For CMS HEP experimentalists, you can use this file: condor-local.jdl

Get your grid certificate & proxy, add CAs to your browser

Get your grid certificate

Follow these instructions to get your personal certificate from CERN and register with the CMS VO. To complete your registration to the CMS VO, you will also need a CERN lxplus account. This process can take up to a week. The same browser on the same machine must be used for all steps connected with getting your grid certificate.

Certificates expire in a year, you will be emailed with instructions on how to get a new one. Follow the instructions below every time you receive a new certificate.

Export your grid certificate from your browser, the interface varies from browser to browser. The exported file will probably have the extension .p12. Copy this file to the cluster following these instructions. Be sure to modify the permissions on the file so it's readable only by you:

chmod 600 YourCert.p12

Now extract your certificate and encrypted private key:

If your first time: mkdir $HOME/.globus

or to replace existing cert with new: rm $HOME/.globus/userkey.pem

openssl pkcs12 -in YourCert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem

openssl pkcs12 -in YourCert.p12 -nocerts -out $HOME/.globus/userkey.pem

chmod 400 $HOME/.globus/userkey.pem

chmod 600 $HOME/.globus/usercert.pem

chmod go-rx $HOME/.globus

You may wish to consider deleting YourCert.p12 from /home, either way, guard this file carefully. To run grid jobs using CRAB, you will also need to register your grid identity in SiteDB following the instructions in this Twiki.

Get your proxy:

Please see the CRAB instructions below for a full recommendation on order of command usage for CRAB jobs.

voms-proxy-init -voms cms

Proxies automatically expire in 24 hours, simply issue this command again to renew it. You can get information on your proxy by issuing the command:

voms-proxy-info -all

Sending proxy in a condor script

Solution from Young Ho

voms-proxy-init -voms cms

cp /tmp/x509up_u`id -u` ~/

Then you can add a line in your .jdl explicitly telling condor where to look for the proxy file:

x509userproxy = /home/yhshin/x509up_u1112

Add CAs to your browser:

When attempting to visit a certificate-enabled website in your browser, you may be warned that the connection is untrusted. This is because the certificate info being presented to you by the site doesn't come from what's known as a "certificate authority" (CA) which you've told your browser you trust. You can choose to continue (in IE) or add an exception (in Firefox), but you'll have to do so repeatedly. This is also considered poor security practice.

While it's not required, you can add trusted CAs to your browser. Then any site presenting certificate info to you from a CA you trust won't show that annoying "are you sure you wish to continue/add exception" window. To add CAs to your browser, navigate to TACAR and click the "install" link underneath the CAs you want to trust. This is an important security decision, so don't add a CA you haven't heard of. The CAs you will want to add for most sites in CMS are: CERN Intermediate, CERN Root, DOEGrids, and ESnet Root - especially DOEGrids.

Submit jobs to the grid using CRAB

These instructions assume you have already gotten your grid certificate. Additionally, you need to have registered your grid certificate in SiteDB. You should already have a CMSSW config file which you've run successfully using interactive commands (cmsRun). An example CMSSW config file is provided in the example if you prefer. This CRAB tutorial can help you get started if you are unfamiliar with CRAB.

• Setup your environment

• Edit crab.cfg

• Edit crab.cfg for SE output

• Create, submit, watch, and retrieve your jobs

• Example

• Submit a CRAB job to the UMD hepcms cluster

Setup your environment:

You no longer need to get the gLite-UI (user interface) environment - step removed.

Get your proxy:

voms-proxy-init -voms cms

Navigate to the CMSSW_X_Y_Z/src you wish to use. Get the CMSSW environment variables:

cmsenv

Get the CRAB environment:

source  /cvmfs/cms.cern.ch/crab/crab.csh

NOTE: You can see this by doing ls -alrth /cvmfs/cms.cern.ch/crab3/crab.csh)

"source /cvmfs/cms.cern.ch/crab/crab.csh" is not supported anymore with CRAB3

Navigate to the subdirectory which contains your CMSSW config file that you wish to submit, and copy the default crab configuration:

cp $CRABPATH/crab.cfg .

Edit crab.cfg:

Edit at least the following values in crab.cfg:

• • scheduler = remoteGlidein : The current only supported scheduler (July 2013)

  • datasetpath: set to a DBS dataset name or leave as none if this is a production job

  • pset: set to the name of your CMSSW config file that you wish to submit

  • total_number_of_events: the minimum number of events you wish to run over or produce (CRAB will run over at least this many events, but may run over more), set to -1 if you wish to run over all the events in the DBS dataset

  • number_of_jobs: the approximate number of jobs to create (a good rule of thumb is that jobs run for ~1 hour), CRAB will attempt to create this many jobs, but may create more or less

  • output_file: a comma separated list of the names of output files that you wish returned to you; typically root files

Edit crab.cfg for SE output:

If your output files might exceed 50MB for each job, then your must stage your output back to a storage element (SE). You can send it back to the hepcms cluster or you can send it to your user area in another SE (typically FNAL for people affiliated with UMD). Although you will stage all files in the output_file list back to a SE, the log files, *.stderr and *.stdout, will still be retrieved via normal means (the -getoutputcommand).

To stage your data back to the hepcms SE:

These directions assume that you have already been given user space on the UMD SE. To request SE user space at UMD, email the System administrators with your CERN username, and the output of voms-proxy-info -all. Set the following values in your crab.cfg file:

[USER]

check_user_remote_dir=0

return_data = 0

copy_data = 1

storage_element = T3_US_UMD

user_remote_dir = subdir

Files will show up in /hadoop/store/user/cern-username/subdir. Please note that due to /hadoop duplication, df -h shows twice the amount of storage space available, and we prefer to keep /hadoop at 75% or below in preparation for single node failure. 

Note: if you get the error: crab: Problems trying remote dir check: Missing environment, that means that you need the check_user_remote_dir=0 option enabled in your crab.cfg.

To stage your data back to your user area in the FNAL SE:

These directions assume that you have already been given user space on the FNAL SE. To request SE user space at FNAL, email Rob Harris & the FNAL T1 admins. You should also have an account at the FNAL LPC. Note that data which is staged out to FNAL cannot be easily copied to the UMD hepcms cluster, even if you used CRAB DBS registration. A way to do this easily is under development by the CMS computing group.

Set the following values in your crab.cfg file:

return_data = 0

copy_data = 1

storage_element = T1_US_FNAL_Buffer

user_remote_dir = subdir

Files will show up in /pnfs/cms/WAX/11/store/user/cern-username/subdir, though due to the special nature of FNAL's SE, avoid using normal operations like ls on /pnfs directories.

Create, submit, watch, and retrieve your jobs:

crab -create

crab -submit

crab -status

crab -getoutput

You can also monitor the status of your jobs with the CMS dashboard. Select your name from the drop down menu and you will be provided with a great deal of information in the form of tables and plots on the status of your jobs.

Output files will be stored inside the crab_x_y_z/res directory and in the SE, if you specified this option in your crab.cfg.

Example:

These files together can be used to create and submit CRAB production jobs with output that stages back directly (not using a SE):

• simple.py

• crab.cfg

To run these files, install a CMSSW release and place simple.py and crab.cfg inside any CMSSW_X_Y_Z/src subdirectory. simple.py has been verified to work in CMSSW_3_9_9, but should work in most modern releases of CMSSW. crab.cfg has been verified to work in CRAB_2_7_8_patch1, but it should work in most modern releases of CRAB with minor modifications.

Setup your environment:

cmsrel CMSSW_3_9_9

cd CMSSW_3_9_9/src

# Copy simple.py and crab.cfg here

cmsenv

source /cvmfs/cms.cern.ch/crab/crab.csh

Create the CRAB jobs:

crab -create

Submit them:

crab -submit

Watch for when they complete (this can take anywhere from 15 minutes to several hours):

crab -status

Once at least one job has completed:

crab -getoutput

Output will be inside the crab_x_y_z/res directory and can be viewed with root.

Submit a CRAB job to the UMD hepcms cluster

If you're located remotely, you may want to submit jobs to the hepcms cluster via CRAB rather than Condor. CRAB jobs can be submitted from any computer which has various grid client tools installed, including CRAB. Consult your siteadmin if you do not know the appropriate commands to set up the grid and CRAB environment. You need to set two parameters in your crab.cfg file:

se_white_list = T3_US_UMD

ce_white_list = T3_US_UMD

Sometimes syntax for ce_white_list changes. Other common styles are:

ce_white_list = umd.edu, T3_US_UMD

ce_white_list = umd.edu

If CRAB jobs claim they cannot match, try modifying your white list syntax.

Of course, the hepcms cluster must have the version of CMSSW that you are using installed and must be hosting the data you are attempting to run over (production jobs require no input data). Check here for your dataset name. You may request to have a CMSSW version installed by contacting the System administrators or Nick Hadley and may bring DBS-registered data to the cluster by submitting a PhEDEx request.

Check your disk usage 

du -hs `find . -type d -user username`