Some commonly-used condor comments
ssh cvssp-condor-master
condor_submit submitfile
condor_rm username #### kill all the processes submitted by user
condor_rm 12345 #### kill the process 12345
condor_q
condor_status
condor_q -af holdreason #### If process on hold, check the error
condor_ssh_to_job 12345 #### go to the server where process 12345 was ran
ls
tail -f _condor_stdout ### monitor the print message
Ctrl+C #### quit monitoring
exit #### exit that server
A vanilla submit file example:
universe = vanilla
executable = train.sh
log = $(cluster).$(process).log~
output = $(cluster).$(process).out~
error = $(cluster).$(process).error~
should_transfer_files = YES
#transfer_output_files = Models
request_GPUs = 1
request_CPUs = 2
request_memory = 400
queue 1
This is due to the inconsistency in the line ending/return between Dos/Windows (CRLF) and Unix (LR) systems.
After running again with
condor_submit -allow-crlf-script ACUPreProcess.submit_file. This time the condor server took in the submit file. However, the process was on hold with a new error:
where the executable train.sh is
#!/bin/bash
source /vol/vssp/ucdatasets/abcde/QLENV3/bin/activate
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/vol/vssp/localsoft/External/cudnn/v7.1.4/cuda-9.2/lib64/
cd /user/HS204/MyPythonCodeDir
python3 TrainModel.py
A docker submit file example
universe = docker
# docker_image = nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04
docker_image = registry.eps.surrey.ac.uk/samplernn-pytorch:4659
executable = /vol/vssp/ucdatasets/abcde/QLENV3/bin/python3
arguments = /user/HS204/MyPythonCodeDir/TrainModel.py
environment = "mount=/vol/vssp/smile,/vol/vssp/signsrc,/vol/vssp/ucdatasets,/vol/vssp/xyz12345/mnist"
log = $(cluster).$(process).log~
output = $(cluster).$(process).out~
error = $(cluster).$(process).error~
should_transfer_files = YES
transfer_output_files = Models
request_GPUs = 1
request_CPUs = 2
request_memory = 400
queue 1
To remotely access university HTCondor
1. The first step is to ssh to surrey using:
ssh username@access.eps.surrey.ac.uk
In order to do so, two-step authentication needs to be setup.
2. ssh to condor server:
ssh condor
3. A job can now be submitted to condor.
Example of submitting a Matlab process using Windows system
The executable file is:
#!/bin/bash
exec /opt/bin/matlab -nodisplay -nojvm -nosplash -r \
"run('PreprocessObservationData.m'); quit"
The submit file is:
#
# Example Job for HTCondor
#
####################
# --------------------------------------------
# Executable and its arguments
executable = ACUjob.sh
arguments =
# ---------------------------------------------------
# Docker: Executable might be stated in docker image so can be omitted
# If specified, overrides the one stated in docker image
# Need to specify the docker image to use
universe = vanilla
# -------------------------------------------------
# Input, Output and Log files
log = c$(cluster).p$(process).log
output = c$(cluster).p$(process).out
error = c$(cluster).p$(process).error
should_transfer_files = YES
transfer_input_files= PreprocessObservationData.m
when_to_transfer_output = ON_EXIT
# -------------------------------------
# Requirements for the Job
requirements = ( HasStornext == true )
# --------------------------------------
# Resource requirements
request_GPUs = 0
request_CPUs = 2
request_memory = 8192
# -----------------------------------
# Queue commands
queue 1
If the main entry PreprocessObservationData.m needs some local functions within the current directory, these need to be listed in the transfer_input_files. For instance, a function abc is called, transfer_input_files= PreprocessObservationData.m,Funcs/abc.m Adding the command addpath(genpath('./')) in PreprocessObservationData.m is not working under this condition.
After logging into the condor server, and trying to submit the process, An error come up that
After checking the common hold reasons at https://indico.cern.ch/event/611296/contributions/2604376/attachments/1471164/2276521/TannenbaumT_UserTutorial.pdf
The most possible reason is still due to Badly formatted bash scripts (have Windows instead of Unix line endings), which happened when I have used the windows notepad or word pad to edit the bash scripts. In this case, -allow-crlf-script option is not working.
Solution: convert the CRLF to LR format. For instance, using the online conversion at https://app.execeratics.com/LFandCRLFonline/?l=en
===============================================================
Conda environments on University system for condor
Install miniconda and generate conda environments
Download 64 bits Linux installers from https://docs.conda.io/en/latest/miniconda.html.
chmod +x installation_file
sh installation_file
During the installation, use a directory on assigned university project space.
Generate conda environments
cd /projectspace/conda38/bin
conda create -n p37tf2 python=3.7
conda install -n p37tf2 tensorflow-gpu.....
conda activate p37tf2
The first time to activate an environment, "conda init bash" is needed, followed by restarting the terminal. After activating, the (base) becomes (p37tf2)
Generate a submit file at the python project directory, cd to the project, and submit process to condor
universe = vanilla
executable = /my_project_space/conda38/envs/p37tf2/bin/python3.7
arguments = $ENV(PWD)/TrainModel.py
log = $(cluster).$(process).log~
output = $(cluster).$(process).out~
error = $(cluster).$(process).error~
should_transfer_files = YES
#transfer_output_files = Models
# -------------------------------------
# Requirements for the Job
rank = ( CUDACapability >= 7.5 )
# hasStornext allows fast access to the storage
requirements = (HasStornext == true) && \
(NotProjectOwned == true)
request_GPUs = 1
request_CPUs = 2
request_memory = 4G
queue 1