Pass configuration from job to op
An OP can be reused by different jobs, but sometimes we need to pass different configurations to the OP to run.
To specify the config item, use the 'config_schema' in the @op decorator. Basic type like str, int can be use directly as type.
More complex types like dict can be specified through the 'Field'.
To read the config during runtime, it needs a reversed keyword 'context' as the first parameter in the op function. Other parameters also be used but the context must be the first one.
within the op function, context.op_config will pick up the required config item, which is 'Job Info' in this case.
@op(config_schema={'Job Info': Field(dict)})
def do_something(context):
config = context.op_config['Job Info']
print(config['Name'])
Now the op is ready, how to pass on the config from a job to an op?
Within the @job decorator, simply set the config with a dictionary of config items, and then the config will be used during running the job.
The config dictionary must comply with a certain format.
1. the first item must be 'ops', as this is config for ops
2. the children items of 'ops' are a list of the op names. this is for assigning config to the correct op.
3. under the op item, it must have a hardcoded 'config' item.
4. within the 'config' item, now you can put in whatever dictionary you want to have. here it puts in a 'Job Info' dictionary so it can pass in a whole dictionary instead of individual items separately into an op.
config_for_all_ops = {
'ops':{
'do_something' : {
'config':{
'Job Info': {
'Name' : 'xxx',
'Site' : 'yyy',
}
}
},
'do_something2' : {
'config':{
'Job Info2': {
'Name' : 'ddd',
'Site' : 'zzz',
}
}
}
}
}
@job(config=config_for_all_ops)
def job_do_sth():
do_something()
There are other ways to configure as well but this seems to be a simple way.
For some configurations, e.g. connection details, it may be good to be coded as a resource, and then reference to the resource from the ops.