Condor Configuration

Config Set-up

Local config files are placed at /etc/condor/config.d/

Suspend-able slots configuration is implemented in slot.conf

#adapted from:

#http://research.cs.wisc.edu/htcondor/manual/v7.6/3_5Policy_Configuration.html#SECTION00459600000000000000

# Lie to Condor, to achieve 2 slots for each real slot

# IDEAL_CORE is shorthand for cores not used by system

IDEAL_CORES = ($(DETECTED_CORES)-1)

NUM_CPUS = $(IDEAL_CORES)*2

# There is no good way to tell Condor that the two slots should be treated

# as though they share the same real memory, so lie about how much

# memory we have.

# Note: In this case we are conservative and not lie about our memory

MEMORY = $(DETECTED_MEMORY)

# Slots 1 through IDEAL_CORES are LowPriority and the rest are

# HighPriority

IsHighPrioritySlot = (SlotID > $(IDEAL_CORES) )

# If I am a suspendable slot, my corresponding nonsuspendable slot is

# my SlotID plus $(IDEAL_CORES)

#LowPrioritySlotState = eval(strcat("slot",SlotID-$(IDEAL_CORES),"_State"))

#HighPrioritySlotState = eval(strcat("slot",SlotID+$(IDEAL_CORES),"_State"))

# use $(NUM_CPUS)-SlotID+1 instead as slots tend to be filled in increasing

# order. Suspension should be applied in reverse order. This setup is most

# beneficial when the node is not almost full

HighPrioritySlotState = eval(strcat("slot",$(NUM_CPUS)-SlotID+1,"_State"))

# The above expression looks at slotX_State, so we need to add

# State to the list of slot attributes to advertise.

STARTD_SLOT_ATTRS = $(STARTD_SLOT_ATTRS) State

# For convenience, advertise these expressions in the machine ad.

STARTD_ATTRS = $(STARTD_ATTRS) IsHighPrioritySlot HighPrioritySlotState LowPrioritySlotState

MyHighPrioritySlotIsBusy = \

(HighPrioritySlotState =?= "Claimed" || HighPrioritySlotState =?= "Preempting")

# LowPriority slots are only willing to start jobs if the job is LowPriority

# and the HighPriority slot is idle.

# HighPriority slots are only willing to start if the job is HighPriority

# LowPriority jobs can only run on LowPriority slots

# HighPriority jobs can only run on HighPriority slots

HighPriorityJobMatch = (IsHighPrioritySlot=?=True && IsHighPriorityJob=?=True)

LowPriorityJobMatch = (IsHighPrioritySlot=!=True && IsHighPriorityJob=!=True)

# Set here to allow or disallow certain users on this node

REQUIREMENTS = True

START = \

( $(LowPriorityJobMatch) && RemoteUser=!="sam@privnet" &&\

$(MyHighPrioritySlotIsBusy)=!=True) || \

($(HighPriorityJobMatch))

START = ($(START)) && $(REQUIREMENTS)

# Suspend the LowPriority slot if the other slot is busy; always allow SAM test to run unobstructed

Suspend=\

$(IsHighPrioritySlot)=!=True && $(MyHighPrioritySlotIsBusy)=?=True

CONTINUE = $(SUSPEND) =!= True

# Ask startd to consider SUSPEND instead of PREEMPT

WANT_SUSPEND = True

# override default behavior so that the node does not slip back to owner state

IS_OWNER = False

A. The number of slots/ processes condor runs is determined by the variable NUM_CPUS. Here we set it to twice the number of CPU minus 1.

B. The slots are divided up into "high priority" and "low priority" slots. On high prioirty slots, IsHighPrioritySlot should evaluate to True.

C. Evaluate the state of the corresponding slot (e.g. in 8 core machine, slot2 corresponds to 14-2+1 = slot13, i.e. second last slot. Conversely, in slot13, this will give corresponding slot2)

D. STARTD_SLOT_ATTRS determines which slot variable is broadcasted to other slots. In this case slotX_State is included in ClassAd of all slots. e.g. slot 2 will have ClassAd variable slot9_State as well as slot2_State, among others. This can be checked with: condor_state compute-0-5 -l | grep -i slot

E. Check if the corresponding slot is busy (in claimed or preempting state)

F. This makes sure that jobs with tag IsHighPriorityJob, and only those jobs, are scheduled to run on high priority slots

G. START is the condition for a particular slot to accept a job. For low priority jobs, the condition includes that no high priority job is running in corresponding slot. (or the low priority job will be immediately suspended and misses potentially better match)

H. Here SUSPEND determines the condition for which a slot is suspended. This only happens in a low priority slot, and only when the corresponding high priority slot is claimed. Otherwise, CONTINUE tells the suspended job to resume.

WANT_SUSPEND tells condor to evaluate SUSPEND but not PREEMPT. IS_OWNER is set to identically false so all slots are always in Unclaimed state, and never fall back to Owner state.

Testing

can be found on /home/kakw/CMSSW_7_6_3/src/condor_test

condor_test.jdl

universe = vanilla

Executable = ping_kak.sh

should_transfer_files = NO

Requirements = TARGET.FileSystemDomain == "privnet" && TARGET.Machine =?= "compute-0-5.privnet"

Output=/data/users/kakw/condor_testing_Mar17/Run300317/output_$(cluster)_$(process).stdout

Error=/data/users/kakw/condor_testing_Mar17/Run300317/output_$(cluster)_$(process).stderr

Log=/data/users/kakw/condor_testing_Mar17/Run300317/output_$(cluster)_$(process).log

#+IsHighPriorityJob = True

Queue 4

ping_kak.sh

#!/bin/env bash

while true; do sleep 10; echo ping; done

Uncomment #+IsHighPriorityJob = True for high priority jobs.

High priority jobs and low priority jobs alone should all run like on normal nodes. Though in each case, only the low half or the upper half of of the slots are claimed, the other half should remain idle.

Report abuse