The best way to take advantage of all the cluster CPU is to run jobs using Condor. Long-running interactive jobs that consume memory and CPU on the interactive nodes are highly discouraged and may be killed. Additionally, condor jobs should run for less than 24 hours, especially if users submit more than 50 jobs. Please split your jobs into smaller jobs. The current configuration of the cluster cannot handle multi-core jobs, instead it is 1 CPU per condor job.
Jobs are submitted via condor using the command:
condor_submit jdl_file
Details on other condor commands and possible values in the jdl file can be found in this Condor guide.
Here is a very simple example jdl file which run the script test.sh (the script you would like to run) which is in area /home/$USER}/condor_test/.
Save the lines below in a file, name the condor_test.jdl
# Here is a very simple submission scrip
Universe = vanilla
Executable = /home/$USER}/condor_test/test.sh
Output =/home/${USER}/condor_test/log/output_$(cluster)_$(process).stdout
Error =/home/${USER}/condor_test/log/output_$(cluster)_$(process).stderr
Log =/home/${USER}/condor_test/log/output_$(cluster)_$(process).log
Should_Transfer_Files = YES
When_to_transfer_output = ON_EXIT
Queue
# end of script
Place this file in one of your directories on the cluster. Edit it for your desired output directory, then submit it to Condor:
condor_submit condor_test.jdl
You can watch the status of the jobs:
condor_q
If you are in interactive node 1 (in1) and you submitted your job in hepcms-rubin.umd.edu, you can do:
condor_q -name hepcms-rubin.umd.edu
Automatically re-run hanging condor jobs with built-in timer
Edit expected amount of time a script is supposed to run. If it is running for too long it might mean the script is hanging.
1 # Here is a condor submission script
2 # To automate job removal in case of a hang
3 # retry if run time exceeds 2 hours, restart no more than 5 times
4 # we should all use these in our job description files
5 universe = vanilla
6 Executable = /bin/sleep
7 getenv = True
8 MINUTE = 60
9 periodic_hold = (CurrentTime - JobCurrentStartDate) >= 120 * $(MINUTE)
10 #optional: for rerunning
11 #periodic_remove = (CurrentTime - JobCurrentStartDate) >= 500 * $(MINUTE)
12 periodic_release = NumJobStarts<5
13 Arguments = 100
14 Queue
Automatic re-run failed jobs
try on_exit_remove expression or on_exit_hold
Example:
on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)
on_exit_hold = (ExitBySignal == False) && (ExitCode == 0)
periodic_release = NumJobStarts<5 && (CurrentTime - JobCurrentStartDate) >= 30 * $(MINUTE)
If you use exit $exitcode to propagate exit code in shell script, you should use:
on_exit_hold = (ExitCode == 0)
Refer to HTCondor manual for more details on on_exit_remove and on_exit_hold:
https://htcondor.readthedocs.io/en/stable/man-pages/condor_submit.html
Pace job start time
Pace job start time by 1 second each. If starting all jobs at the same moment causes problem this may be helpful. However, beware that higher delays might take long time for all jobs to start running (e.g. 300jobs * 60 sec = 5 hours), and if your code causes a race condition, it is a design problem and this configuration will not help.
next_job_start_delay=1
Condor Machine Requirement
It is important to specify job requirements so that they make use of the slots that can run them properly. This may be due to lack of mounted disk, OS and software environment, or simply jobs failling to run on specific node. To rule out certain node, use:
Rank = TARGET.Machine =!= "siab-1.umd.edu"
Condor Machine Ranking
Default ranking algorithm is given by:
NEGOTIATOR_PRE_JOB_RANK = (JobOverridesNegotiatorRank =!= True) * \
isUndefined(RemoteOwner) * (KFlops*(-SlotID) +\
regexp("r720.*",machine)*5.0e6 + regexp("r510.*",machine)*1.0e6)
This means jobs will generally be equally distributed to all nodes, but with slight preference to r720 and r510s
This can by overriden by Rank expression in Job description jdl files. For example, to prefer one compute node:
+JobOverridesNegotiatorRank = True
Rank = TARGET.Machine =?= "compute-0-10.privnet"
With regular expressions we can select a group of nodes:
+JobOverridesNegotiatorRank = True
Rank = regexp("compute.*",TARGET.Machine)
Long Running Jobs
Jobs running longer than a day needs to add long running tag or they risk being kicked off. This is to prevent jobs with loop condition to run forever
Multi-thread Jobs
These are jobs that automatically scales with number of available threads in a machine. If they are submitted to normal slots they crowd out resources for other jobs
Email notification for condor jobs
Notify yourself through email. Options are always (all state changes), or complete (only send email when job completes)
1 universe = vanilla
2 Executable = /bin/sleep
3 notify_user = username@umd.edu
4 notification = always
5 Arguments = 100
6 Queue
More Condor job description recipes
Please refer to the official HTCondor manual
http://research.cs.wisc.edu/htcondor/manual/v7.6/condor_submit.html
A Short Introduction to HTCondor
http://personalpages.to.infn.it/~gariazzo/htcondor/concepts.html
Condor Log Analyzer
Condor log analyzer is a handy way to check batch performance after the jobs finish. This is particularly useful if a lot of jobs are submitted
CMSSW (cmsRun) with Condor
The following three files together are an example of how to submit CMSSW jobs to the cluster with Condor:
To run these files, setup a CMSSW release and place condor-simple.py inside the src subdirectory. condor-simple.py has been verified to work in CMSSW_3_9_9, though it should work in most modern releases. To run other software (i.e., Madgraph), you do not need a CMSSW release area. condor-executable.sh and condor-jobs.jdl can be placed together in any directory. Edit condor-jobs.jdl with your desired output directory, a unique prefix name to help you identify the files, and your CMSSW release area. Make condor-executable.sh executable:
chmod +x condor-executable.sh
Then submit the jobs:
condor_submit condor-jobs.jdl
You can monitor the status of your jobs, they should take about 30 minutes to complete once they start running:
condor_q -submitter your_username
You can also watch condor_q:
watch -n 60 "condor_q -submitter your_username"
Once jobs are complete, the *.condor files will indicate if condor itself ran successfully, as well as providing the exit code of the CMSSW job. The *.stdout and *.stderr files will show output from condor-executable.sh and the *.log files will show output from CMSSW itself.
For CMS HEP experimentalists, you can use this file: condor-local.jdl
Follow these instructions to get your personal certificate from CERN and register with the CMS VO. To complete your registration to the CMS VO, you will also need a CERN lxplus account. This process can take up to a week. The same browser on the same machine must be used for all steps connected with getting your grid certificate.
Certificates expire in a year, you will be emailed with instructions on how to get a new one. Follow the instructions below every time you receive a new certificate.
Export your grid certificate from your browser, the interface varies from browser to browser. The exported file will probably have the extension .p12. Copy this file to the cluster following these instructions. Be sure to modify the permissions on the file so it's readable only by you:
chmod 600 YourCert.p12
Now extract your certificate and encrypted private key:
If your first time: mkdir $HOME/.globus
or to replace existing cert with new: rm $HOME/.globus/userkey.pem
openssl pkcs12 -in YourCert.p12 -clcerts -nokeys -out $HOME/.globus/usercert.pem
openssl pkcs12 -in YourCert.p12 -nocerts -out $HOME/.globus/userkey.pem
chmod 400 $HOME/.globus/userkey.pem
chmod 600 $HOME/.globus/usercert.pem
chmod go-rx $HOME/.globus
You may wish to consider deleting YourCert.p12 from /home, either way, guard this file carefully. To run grid jobs using CRAB, you will also need to register your grid identity in SiteDB following the instructions in this Twiki.
Please see the CRAB instructions below for a full recommendation on order of command usage for CRAB jobs.
voms-proxy-init -voms cms
Proxies automatically expire in 24 hours, simply issue this command again to renew it. You can get information on your proxy by issuing the command:
voms-proxy-info -all
Solution from Young Ho
voms-proxy-init -voms cms
cp /tmp/x509up_u`id -u` ~/
Then you can add a line in your .jdl explicitly telling condor where to look for the proxy file:
x509userproxy = /home/yhshin/x509up_u1112
When attempting to visit a certificate-enabled website in your browser, you may be warned that the connection is untrusted. This is because the certificate info being presented to you by the site doesn't come from what's known as a "certificate authority" (CA) which you've told your browser you trust. You can choose to continue (in IE) or add an exception (in Firefox), but you'll have to do so repeatedly. This is also considered poor security practice.
While it's not required, you can add trusted CAs to your browser. Then any site presenting certificate info to you from a CA you trust won't show that annoying "are you sure you wish to continue/add exception" window. To add CAs to your browser, navigate to TACAR and click the "install" link underneath the CAs you want to trust. This is an important security decision, so don't add a CA you haven't heard of. The CAs you will want to add for most sites in CMS are: CERN Intermediate, CERN Root, DOEGrids, and ESnet Root - especially DOEGrids.
These instructions assume you have already gotten your grid certificate. Additionally, you need to have registered your grid certificate in SiteDB. You should already have a CMSSW config file which you've run successfully using interactive commands (cmsRun). An example CMSSW config file is provided in the example if you prefer. This CRAB tutorial can help you get started if you are unfamiliar with CRAB.
You no longer need to get the gLite-UI (user interface) environment - step removed.
Get your proxy:
voms-proxy-init -voms cms
Navigate to the CMSSW_X_Y_Z/src you wish to use. Get the CMSSW environment variables:
cmsenv
Get the CRAB environment:
source /cvmfs/cms.cern.ch/crab/crab.csh
NOTE: You can see this by doing ls -alrth /cvmfs/cms.cern.ch/crab3/crab.csh)
"source /cvmfs/cms.cern.ch/crab/crab.csh" is not supported anymore with CRAB3
Navigate to the subdirectory which contains your CMSSW config file that you wish to submit, and copy the default crab configuration:
cp $CRABPATH/crab.cfg .
Edit at least the following values in crab.cfg:
scheduler = remoteGlidein : The current only supported scheduler (July 2013)
datasetpath: set to a DBS dataset name or leave as none if this is a production job
pset: set to the name of your CMSSW config file that you wish to submit
total_number_of_events: the minimum number of events you wish to run over or produce (CRAB will run over at least this many events, but may run over more), set to -1 if you wish to run over all the events in the DBS dataset
number_of_jobs: the approximate number of jobs to create (a good rule of thumb is that jobs run for ~1 hour), CRAB will attempt to create this many jobs, but may create more or less
output_file: a comma separated list of the names of output files that you wish returned to you; typically root files
If your output files might exceed 50MB for each job, then your must stage your output back to a storage element (SE). You can send it back to the hepcms cluster or you can send it to your user area in another SE (typically FNAL for people affiliated with UMD). Although you will stage all files in the output_file list back to a SE, the log files, *.stderr and *.stdout, will still be retrieved via normal means (the -getoutputcommand).
To stage your data back to the hepcms SE:
These directions assume that you have already been given user space on the UMD SE. To request SE user space at UMD, email the System administrators with your CERN username, and the output of voms-proxy-info -all. Set the following values in your crab.cfg file:
[USER]
check_user_remote_dir=0
return_data = 0
copy_data = 1
storage_element = T3_US_UMD
user_remote_dir = subdir
Files will show up in /hadoop/store/user/cern-username/subdir. Please note that due to /hadoop duplication, df -h shows twice the amount of storage space available, and we prefer to keep /hadoop at 75% or below in preparation for single node failure.
Note: if you get the error: crab: Problems trying remote dir check: Missing environment, that means that you need the check_user_remote_dir=0 option enabled in your crab.cfg.
To stage your data back to your user area in the FNAL SE:
These directions assume that you have already been given user space on the FNAL SE. To request SE user space at FNAL, email Rob Harris & the FNAL T1 admins. You should also have an account at the FNAL LPC. Note that data which is staged out to FNAL cannot be easily copied to the UMD hepcms cluster, even if you used CRAB DBS registration. A way to do this easily is under development by the CMS computing group.
Set the following values in your crab.cfg file:
return_data = 0
copy_data = 1
storage_element = T1_US_FNAL_Buffer
user_remote_dir = subdir
Files will show up in /pnfs/cms/WAX/11/store/user/cern-username/subdir, though due to the special nature of FNAL's SE, avoid using normal operations like ls on /pnfs directories.
crab -create
crab -submit
crab -status
crab -getoutput
You can also monitor the status of your jobs with the CMS dashboard. Select your name from the drop down menu and you will be provided with a great deal of information in the form of tables and plots on the status of your jobs.
Output files will be stored inside the crab_x_y_z/res directory and in the SE, if you specified this option in your crab.cfg.
These files together can be used to create and submit CRAB production jobs with output that stages back directly (not using a SE):
To run these files, install a CMSSW release and place simple.py and crab.cfg inside any CMSSW_X_Y_Z/src subdirectory. simple.py has been verified to work in CMSSW_3_9_9, but should work in most modern releases of CMSSW. crab.cfg has been verified to work in CRAB_2_7_8_patch1, but it should work in most modern releases of CRAB with minor modifications.
Setup your environment:
cmsrel CMSSW_3_9_9
cd CMSSW_3_9_9/src
# Copy simple.py and crab.cfg here
cmsenv
source /cvmfs/cms.cern.ch/crab/crab.csh
Create the CRAB jobs:
crab -create
Submit them:
crab -submit
Watch for when they complete (this can take anywhere from 15 minutes to several hours):
crab -status
Once at least one job has completed:
crab -getoutput
Output will be inside the crab_x_y_z/res directory and can be viewed with root.
If you're located remotely, you may want to submit jobs to the hepcms cluster via CRAB rather than Condor. CRAB jobs can be submitted from any computer which has various grid client tools installed, including CRAB. Consult your siteadmin if you do not know the appropriate commands to set up the grid and CRAB environment. You need to set two parameters in your crab.cfg file:
se_white_list = T3_US_UMD
ce_white_list = T3_US_UMD
Sometimes syntax for ce_white_list changes. Other common styles are:
ce_white_list = umd.edu, T3_US_UMD
ce_white_list = umd.edu
If CRAB jobs claim they cannot match, try modifying your white list syntax.
Of course, the hepcms cluster must have the version of CMSSW that you are using installed and must be hosting the data you are attempting to run over (production jobs require no input data). Check here for your dataset name. You may request to have a CMSSW version installed by contacting the System administrators or Nick Hadley and may bring DBS-registered data to the cluster by submitting a PhEDEx request.
du -hs `find . -type d -user username`