Joblib Example

You can always read more detail on official page https://joblib.readthedocs.io/en/latest/

Affinity issue

On Greene, you have to do this trick to address problem with affinity (more info)

Start Python

python3

Execute the following in your Python session

## see current affinityimport osos.sched_getaffinity(0)## we see only one CPU# reset affinity - read about affinity in man tasksetos.system("taskset -p 0xFFFFFFFF %d" % os.getpid())# checkos.sched_getaffinity(0)

Quit Python

exit()

Examples to run on Greene:

Collection of examples is available on the developer's website:

https://joblib.readthedocs.io/en/latest/auto_examples/index.html#parallel-examples

Parallel Processes

This will run on several CPUs inside one node

You can pass as a parameter only objects which can be pickled. Or use this approach

def myfunc: i*2Parallel(n_jobs=2)(delayed(myfunc)(i) for i in range(10))

Parallel Threads

This will run using many thread on one CPU only

You can pass any objects present in Python. There is no need to pickle them.

def myfunc: i*2Parallel(n_jobs=2, prefer="threads")(delayed(myfunc)(i) for i in range(10))