A proper way to have sophisticated schedules in Databricks is using the CRON expression (Cron Trigger Tutorial )
A CRON expression is a line of plain text containing 6 compulsory fields and 1 optional field delimited by space:
Seconds Minutes Hours Day_of_Month Month Day_of_Week Year
The first 6 fields are compulsory and the 'Year' is optional. Below explains how it works with examples:
* 0,30 * * * ?
It means every minute 0 and minute 30 (i.e. every 30 minutes). The * means all values for the corresponding field. The 0,30 means minutes 0 and 30 delimited by comma. The ? means no specific value for Day_of_Week which will be explained later below.
* 0/5 * * * ?
It means every 5 minutes starting from minute 0. The “0/5“ means a range starting from minute 0 and the increment interval is 5 minutes, so it represents 0, 5, 10, 15 .. 55
* 0/5 7-9 1 * ?
It means every 5 minutes between 7am to 9am on the first day of every month. The “7-9“ means a range from 7 to 9 (equivalent to 7,8,9). The Day_of_month is set to 1 so it is the first day of the month.
If need to specify the last day of the month, which can be 28th or 30th or 31st, simply use “L“ to represent the last day, so it becomes * 0/5 7-9 L * ?
if need to specify the last week of the month, use “LW“ to represent the last week, so it becomes * 0/5 7-9 LW * ?
* 0/5 7-9 ? * 1
It means every 5 minutes between 7am to 9am on the first day of every week. Note here the question mark “?“ is set to the Day_of_Month field, which means no specific value. Day_of_Month and Day_of_Week are two conflicting fields. If the day of month is set, it can’t guarantee the day of week, and vice versa. Therefore, one of the two fields needs to be “?“ to make sense.
* 0/5 * ? * 2#1
It means the every 5 minutes in the first Monday of the month. The first digit “2“ of “2#1“ specifies the day of the week. Note day 1 is Sunday and day 2 is Monday, and so on. The “#1“ means the first one in the month.
0 0/5 * ? * 2-6 2022-2030
It means every 5 minutes on weekdays and from year 2022 to 2030. Note the optional year field is used to specify an end for the schedule. Also the Seconds field is set to 0 so it’s triggered at exactly second 0 of the minute, otherwise it may be triggered multiple times within the minute.